PIQA: Reasoning about Physical Commonsense in Natural Language
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, Yejin Choi
Introduction
Before children learn language, they already start forming categories and concepts based on the physical properties of objects around them (?). This model of the world grows richer as they learn to speak, but already captures physical commonsense knowledge about everyday objects: their physical properties, affordances, and how they can be manipulated. This knowledge is critical for day-to-day human life, including tasks such as problem solving (what can I use as a pillow when camping?) and expressing needs and desires (bring me a harder pillow). Likewise, we hypothesize that modeling physical commonsense knowledge is a major challenge on the road to true AI-completeness, including robots that interact with the world and understand natural language.
Much of physical commonsense can be expressed in language, as the versatility of everyday objects and common concepts eludes other label schemes. However, due to issues of reporting bias, these commonsense properties - facts like ‘it is a bad idea to apply eyeshadow with a toothpick’ are rarely directly reported. Although much recent progress has been made in Natural Language Processing through a shift towards large-scale pretrained representations from unlabeled text (?; ?; ?), the bulk of the success of this paradigm has been on core abstract tasks and domains. State-of-the-art models can reliably answer questions given an encyclopedia article (?) or recognize named entities (?), but it is not clear whether they can robustly answer questions that require physical commonsense knowledge.
To study this question and begin bridging the representational gap, we introduce Physical Interaction: Question Answering, or PIQA to evaluate language representations on their knowledge of physical commonsense. We focus on everyday situations with a preference for atypical solutions. Our dataset is inspired by instructables.com, which provides users with instructions on how to build, craft, bake, or manipulate objects using everyday materials. We asked annotators to provide semantic perturbations or alternative approaches which are otherwise syntactically and topically similar to ensure physical knowledge is targeted. The dataset is further cleaned of basic artifacts using the AFLite algorithm introduced in (?; ?) which is an improvement on adversarial filtering (?; ?).
Throughout this work we first detail the construction of our new benchmark for physical commonsense. Second, we show that popular approaches to large-scale language pretraining, while highly successful on many abstract tasks, fall short when a physical model of the world is required. Finally, our goal is to elicit further research into building language representations that capture details of the real world. To these ends, we perform error and corpora analyses to provide insights for future work.
Dataset
We introduce a new dataset, PIQA , for benchmarking progress in physical commonsense understanding. The underlying task is multiple choice question answering: given a question and two possible solutions , a model or a human must choose the most appropriate solution, of which exactly one is correct. We collect data with how-to instructions as a scaffold, and use state-of-the-art approaches for handling spurious biases, which we will discuss below.
Our goal is to construct a resource that requires concrete physical reasoning. To achieve this, we provide a prompt to the annotators derived from instructables.com. The instructables website is a crowdsourced collection of instructions for doing everything from cooking to car repair. In most cases, users provide images or videos detailing each step and a list of tools that will be required. Most goals are simultaneously rare and unsurprising. While an annotator is unlikely to have built a UV-Flourescent steampunk lamp or made a backpack out of duct tape, it is not surprising that someone interested in home crafting would create these, nor will the tools and materials be unfamiliar to the average person. Using these examples as the seed for their annotation, helps remind annotators about the less prototypical uses of everyday objects. Second, and equally important, is that instructions build on one another. This means that any QA pair inspired by an instructable is more likely to explicitly state assumptions about what preconditions need to be met to start the task and what postconditions define success.
Collecting data through goal-solution pairs
Unlike traditional QA tasks, we define our dataset in terms of Goal and Solution pairs (see Figure 2 for example Goal-Solution pairs and types of physical reasoning). The Goal in most cases can be viewed as indicating a post-condition and the solutions indicate the procedure for accomplishing this. The more detailed the goal, the easier it is for annotators to write both correct and incorrect solutions. As noted above, the second component of our annotation design is reminding people to think creatively. We initially experimented with asking annotators for (task, tool) pairs via unconstrained prompts, but found that reporting bias swamped the dataset. In particular, when thinking about how to achieve a goal, people most often are drawn to prototypical solutions and look for tools in the kitchen (e.g. forks and knives) or the garage (e.g. hammers and drills). They rarely considered the literal hundreds of other everyday objects that might be in their own homes (e.g. sidewalk chalk, shower curtains, etc).
To address this, and flatten the distribution of referenced objects (see Figure 5), we prompt the annotations with links to instructables. Specifically, annotators were asked to glance at the instructions of an instructable and pull out or have it inspire them to construct two component tasks. They would then articulate the goal (often centered on atypical materials) and how to achieve it. In addition, we asked them to provide a permutation to their own solution which makes it invalid, often subtly (Figure 2). To further assist diversity we seed annotators with instructables drawn from six categories (costume, outside, craft, home, food, and workshop). We asked that two examples be drawn per instructable to encourage one of them to come later in the process and require precise articulation of pre-conditions.
During validation, examples with low agreement were removed from the data. This often meant that correct examples were removed that required expert level knowledge of a domain (e.g. special woodworking terminology) which should not fall under the umbrella of “commonsense.” Because, we focus on human generated tricks, annotators were free to come up with clever ways to hide deception. Often, this meant making very subtle changes to the solution to render it incorrect. In these cases, the two solutions may differ by as little as one word. We found that annotations used both simple linguistic tricks (e.g. negation and numerical changes) and often swapped a key action or item for another that was topically similar but not helpful for completing the given goal. For this reason, our interface also includes a diff button which highlights where the solutions differ. This improved annotator accuracy and speed substantially. Annotator pay averaged $/hr according to both self-reporting on turkerview.com and our timing calculations.
Statistics
In total our dataset is comprised of over 16,000 training QA pairs with an additional 2K and 3k held out for development and testing, respectively. Our goals, as tokenized by Spacy,https://spacy.io – all data was collected in English. average 7.8 words and both correct and incorrect solutions average 21.3 words. In total, this leads to over 3.7 million lexical tokens in the training data.
Figure 4 shows a plot of the correct and incorrect sequence lengths (as tokenized by the GPT BPE tokenizer), with the longest 1% of the data removed. While there are minor differences, the two distributions are nearly identical.
We also analyzed the overlap in the vocabulary and find that in all cases (noun, verb, adjective, and adverb) we see at least an 85% overlap between words used in correct and incorrect solutions. In total we have 6,881 unique nouns, 2,493 verbs, 2,263 adjectives, and 604 adverbs in the training data.. The most common of each are plotted in Figure 5 alongside their cumulative distributions. Again, this helps verify that the dataset revolves very heavily around physical phenomena, properties, and manipulations. For example, the top adjectives include state (dry, clean, hot) and shape (small, sharp, flat); adverbs include temporal conditions (then, when) and manner (quickly, carefully, completely). These properties often differentiate correct from incorrect answers, as shown in examples throughout the paper. We also color words according to their concreteness score (?), though many “abstract” words have concrete realizations in our dataset.
Removing Annotation Artifacts
As noted previously, we use AFLite (?) to remove stylistic artifacts and trivial examples from the data, which have been shown to artificially inflate model performance on previous NLI benchmarks (?; ?). The AFLite algorithm performs a systematic data bias reduction: it discards instances whose given feature representations are collectively highly indicative of the target label. In practice, we use 5,000 examples from the original dataset to fine-tune BERT-Large for this task and compute the corresponding embeddings of all remaining instances. AFLite uses an ensemble of linear classifiers trained on random subsets of the data to determine whether these pre-computed embeddings are strong indicators of the correct answer option. Instead of having to specifically identify the possible sources of biases, this approach enables unsupervised data bias reduction by relying on state-of-the-art methods to uncover undesirable annotation artifacts. For more information about AFLite, please refer to (?).
Experiments
In this section, we test the performance of state-of-the-art natural language understanding models on our dataset, PIQA. In particular, we consider the following three large-scale pretrained transformer models:
GPT (?) is a model that processes text left-to-right, and was pretrained using a language modeling objective. We use the original 124M parameter GPT model.
BERT (?) is a model that process text bidirectionally, and thus was pretrained using a special masked language modeling objective. We use BERT-Large with 340M parameters.
RoBERTa (?) is a version of the BERT model that was made to be significantly more robust through pretraining on more data and careful validation of the pretraining hyperparameters. We use RoBERTa-Large, which has 355M parameters.
We follow standard best practices in adapting these models for two-way classification. We consider the two solution choices independently: for each choice, the model is provided the goal, the solution choice, and a special [CLS] token. At the final layer of the transformer, we extract the hidden states corresponding to the positions of each [CLS] token. We apply a linear transformation to each hidden state and apply a softmax over the two options: this approximates the probability that the correct solution is option A or B. During finetuning, we train the model using a cross-entropy loss over the two options. For GPT, we follow the original implementation and include an additional language modeling loss, which improved training stability.
Generally, we found that finetuning was often unstable with some hyperparameter configurations leading to validation performance around chance, particularly for BERT. We follow best practices in using a grid search over learning rates, batch sizes, and the number of training epochs for each model, and report the best-scoring configuration as was found on the validation set. For all models and experiments, we used the transformers library and truncated examples at 150 tokens, which affects 1% of the data.
Manual inspection of the development errors show that some “mistakes” are actually correct but required a web-search to verify. Human performance was calculated by a majority vote. Annotators were chosen to participate that achieved 90% on the qualification HIT from before. It is therefore, completely reasonable that automated methods trained on large web crawls may eventually surpass human performance here. Human evaluation was performed on development data, and the train, development, and test folds were automatically produced by AFLite.
We present our results in Table 1. As the dataset was constructed to be adversarial to BERT, it is not surprising that it performs the worst of three models despite generally outperforming GPT on most other benchmarks. Comparing GPT and RoBERTa we see that despite more training data, a larger vocabulary, twice the number of parameters and careful construction of robust training, there is only a 8pt performance gain and RoBERTa still falls roughly 18 points short of human performance on this task. As noted throughout, exploring this gap is precisely the purpose for PIQA existing and which facets of the dataset fool RoBERTa is the focus of the remainder of this paper.
Analysis
In this section, we unpack the results of state-of-the-art models on PIQA. In particular, we take a look at the errors made by the top-performing model RoBERTa, as a view towards the physical commonsense knowledge that can be learned through language alone.
The setup of PIQA allows us to use it to probe the inner workings of deep pretrained language models, and to determine the extent of their physical knowledge. In this way, our dataset can augment prior work on studying to what extent models such as BERT understand syntax (?). However, while syntax is a well studied problem within linguistics, physical commonsense does not have as rich a literature to borrow from, making its dimensions challenging to pin down.
Simple concepts. Understanding the physical world requires a deep understanding of simple concepts, such as “water” or “ketchup,” and their affordances and interactions with respect to other concepts. Though our dataset covers interactions between and with common objects, we can analyze the space of concepts in the dataset by performing a string alignment between solution pairs. Two solution choices that differ by editing a single phrase must by definition test the commonsense understanding of that phrase.
In Figure 6 we show the distribution of the edit distance between solution choices. We compute edit distance over tokenized and lowercased strings with punctuation removed. We use a cost of 1 for edits, insertions, and deletions. Most of the dataset covers simple edits between the two solution choices: roughly 60% of the dataset in both validation and training involves a 1-2 word edit between solutions. In the bottom of Figure 6, we show that the dataset complexity generally increases with the edit distance between the solution pairs. Nevertheless, the head of the distribution represents a space that is simple to study.
Single-word edits. In Figure 7, we plot the accuracy of RoBERTa among dataset examples that differ by a single word. More formally, we consider examples whereby moving from to , or vice versa, requires editing a given word .We additionally allow for an additional insertion; this helps to capture simple phrases like going from ‘water’ to ‘olive oil.’ Nevertheless, these multiword expressions tend to be less common, which is why we omit them in Figure 7. We show examples of words that occur frequently in both the training and validation splits of the dataset, which allows RoBERTa to refine representations of these concepts during training and gives us a large enough sample size to reliably estimate model performance.
As shown, RoBERTa struggles to understand certain highly flexible relations. In particular, Figure 7 highlights the difficulty of correctly answering questions that differ by the words ‘before,’ ‘after’, ‘top‘, and ‘bottom’: RoBERTa performs nearly at chance when encountering these.
Interestingly, the concepts shown in Figure 7 suggest that RoBERTa also struggles to understand many common, more versatile, physical concepts. Though there are 300 training examples wherein the solution choices differ by the word ‘water.’ RoBERTa performs worse than average on these replacements. On the other hand, RoBERTa does much better at certain nouns, such as ‘spoon.’
Common replacements in PIQA. We dig into this further in Figure 8, where we showcase the most common replacements for three examples: ‘water,’ ‘spoon,’ and ‘freeze.’ While ‘water’ is prevalent in the training set, it is also highly versatile. One can try to substitute it with a variety of different household items, such as ‘milk’ or ‘alcohol,’ often to disastrous effects. However, ‘spoons’ have fewer challenging properties. A spoon cannot generally be substituted with a utensil that is sharp or has prongs, such as a fork, a knife, or a toothpick. RoBERTa obtains high accuracy on ‘spoon’ examples, which suggests that it might understand this simple affordance, but does not capture the long tail of affordances associated with ‘water.’
Qualitative results
Our analysis thus far has been on simple-to-analyze single word expressions, where we have shown that the state-of-the-art language model, RoBERTa, struggles at a nuanced understanding of key commonsense concepts, such as relations. To further probe the knowledge gap of these strong models, we present qualitative examples in Figure 9. The examples are broadly representative of larger patterns: RoBERTa can recognize clearly ridiculous generations (Figure 9, top left) and understands differences between some commonsense concepts (bottom left). It’s important to note, that in both cases the correct answer is prototypical and something we might expect the models to have seen before.
However, it struggles to tell the difference between subtle relations such as top and bottom (top right of Figure 9). Moreover, it struggles with identifying non-prototypical situations (bottom right). Though using a gluestick as feet for a coaster is uncommon, to a human familiar with these concepts we can visualize the action and its result to verify that the goal has been achieved. Overall, these examples suggest that physical understanding – particularly involving novel combinations of common objects – challenges models that were pretrained on text only.
Related Work
Physical understanding is broad domain that touches on everything from scientific knowledge (?) to the interactive acquisition of knowledge by embodied agents (?). To this end, work related to the goals of our benchmark span the NLP, Computer Vision and Robotics communities.
Language. Within NLP, in addition to large scale models, there has also been progress on reasoning about cause and effect effects/implications within these models (?), extracting knowledge from them (?), and investigating where large scale language models fail to capture knowledge of tools and elided procedural knowledge in recipes (?). The notion of procedural knowledge and instruction following is a more general related task within vision and robotics. From text alone, work has shown that much can be understood about the implied physical situations of verb usage (?) and relative sizes of objects (?).
Vision. Physical knowledge can be discovered and evaluated within the visual world. Research has studied predicting visual relationships in images (?) and as well as actions and their dependent objects (?). Relatedly, the recent HAKE dataset (?) specifically annotates which object/body-parts are essential to completing or defining an action. Image data also allows for studying the concreteness of nouns and provides a natural path forward for further investigation (?). Related to physical commonsense, research in visual commonsense has studied intuitive physics (?), cause-effect relationships (?), and what can be reasonably inferred beyond a single image (?).
Robotics. Learning from interaction and intuitive physics (?) can also be encoded as priors when exploring the world (?) and internal models of physics, shape, and material strength enable advances in tool usage (?) or construction (?). Key to our research aims in this work is helping to build language tools which capture enough physical knowledge to speed up the bootstrapping of robotic-language applications. Language tools should provide strong initial priors for learning (?; ?) that are then refined through interaction and dialogue (?).
Conclusion
We have evaluated against large-scale pretrained models as they are in vogue as the de facto standard of progress within NLP, but are primarily interested in their performance and failings as a mechanism for advancing the position that learning about the world from language alone, is limiting. Future research, may “match” humans on our dataset by finding a large source of in-domain data and fine-tuning heavily, but this is very much not the point. Philosophically, knowledge should be learned from interaction with the world to eventually be communicated with language.
In this work we introduce the Physical Interaction: Question Answering or PIQA benchmark for evaluating and studying physical commonsense understanding in natural language models. We find the best available pretrained models lack an understanding of some of the most basic physical properties of the world around us. Our goal with PIQA is to provide insight and a benchmark for progress towards language representations that capture knowledge traditionally only seen or experienced, to enable the construction of language models useful beyond the NLP community.
Acknowledgements
We thank the anonymous reviewers for their insightful suggestions. This research was supported in part by NSF (IIS-1524371, IIS-1714566), DARPA under the CwC program through the ARO (W911NF-15-1-0543), DARPA under the MCS program through NIWC Pacific (N66001-19-2-4031), and the NSF-GRFP No. DGE-1256082. Computations on beaker.org were supported in part by Google Cloud.