Do Neural Language Representations Learn Physical Commonsense?

Introduction

Understanding everyday natural language communication requires a rich spectrum of physical commonsense knowledge. Consider the example dialog sketched in Figure 1. A simple observation that, “The blender is broken again!” triggers myriad pieces of implied understanding (e.g., that something which requires electricity will only work with a source of power). Such knowledge is rarely stated explicitly (?, ?), and instead can be inferred on-the-fly as needed.

In this paper, we study physical commonsense knowledge underlying natural language understanding, organized as interactions among three distinct concepts: (i) objects, (ii) their attributes (properties), and (iii) the actions that can be applied to them (affordances) (Figure 1, bottom). The premise of our study is that language models trained on a sufficiently large amount of text can recover a great deal of physical commonsense knowledge about each of these concepts. However, aspects of this knowledge may only be implicit in natural language utterances. For example, answering a question from the Winograd Schema Challenge (?, ?)—”The trophy would not fit in the brown suitcase because it was too big. What was too big?”—implicitly requires the physical commonsense reasoning that “in order to fit X in Y, X should be relatively smaller compared to Y,” which essentially requires reasoning about the affordances of objects (fit X in Y) from their attributes (relative sizes of X and Y).

We investigate the extent to which neural language models trained on a massive amount of text demonstrate various aspects of physical commonsense knowledge and reasoning. Our analysis includes word embeddings such as GloVe (?, ?), as well as more recent contextualized representations like ELMo (?, ?) and BERT (?, ?). Such models are trained without supervision by exposing them to billions of words, and allowing them to extract patterns purely from token prediction tasks that can be derived directly from raw text. These language representation models have established unprecedented performance on a wide range of evaluations, including natural language inference and commonsense reasoning.

How much do these large, unsupervised models of language learn about physical commonsense knowledge? Some recent work has studied the capabilities of word embeddings to predict an object’s properties (?, ?, ?). Motivated by these efforts to understand language representations, we present several contributions. We propose two datasets: the abstract dataset, a refreshed version of the McRate dataset (?, ?), pruned and densely annotated to eliminate false negatives present in previous work; and the situated dataset, with annotations for objects’ properties and affordances in real-world images sampled from the MS COCO dataset (?, ?). As in previous work, we consider the prediction task of linking objects and their properties (O$\longleftrightarrow$P), but with our new situated dataset, we are also able to study the connection between objects and their affordances (O$\longleftrightarrow$A), as well as between affordances and properties (A$\longleftrightarrow$P). We also study the latest models from the natural language processing community (ELMo, BERT) using in-context word representations, and present results for all of our proposed datasets and tasks. Our analysis suggests that current neural language representations are proficient at guessing the affordances and properties of objects, but lack the ability to reason about the relationship between affordances and properties itself.

Characterizing Objects through Properties and Affordances

We use the term properties to refer to the static characteristics of objects. They encompass our commonsense understanding of what something is like. For example, we might say that an apple has the property of being edible, or that a plant is stationary.

As with ? (?), properties capture the general perception of a thing. Exceptions naturally arise. For example, specific instances can violate the general properties of an object, such as the inediblilty of a rotten apple. Additionally, subtypes can diverge from the exemplar of a category: the Venus flytrap is a plant with the ability to move.

Affordances

We express an object’s actions with verbs. One way to focus on understanding the actions of objects is to focus on their affordances. Coined by ? (?), this term initially described animal-perceived uses for an object, but has since come to mean the perceived uses of an object in a given environment (?, ?, ?).

Here, we take a simpler, human-centric definition. We consider an object’s affordances to be, “what actions do humans take with an object?” For example, boots commonly afford wear, kick off, lace up, and put on.

Inference Between Affordances and Properties

Affordances and properties exhibit a surprising connection. As humans, we are able to infer many of an object’s affordances based on its properties (A$\leftarrow$P). The same is also true in the reverse (A$\rightarrow$P).

Consider an exchange: “You think you could fit that boulder in your truck?” “No way! That thing was so big you could go for a hike on it.” We might sketch out some of this information as:

While the above information only concerns a property’s relative value (comparative size), a broad range of information can be inferred between affordances and properties. Our focus in this work is on absolute properties, for example:

The implications ($\implies$) should be taken with a probabilistic grain of salt. However, they capture our intuitions about what we expect to be true. Wouldn’t it be surprising to shatter something that isn’t rigid, or plug-in something that doesn’t take power?

Humans use the link between affordances and properties to recover information. Can machine learning models do the same? It is is difficult to model these implications based on text alone because there is no direct evidence for the implied information. Any implication that can be trivially understood by a person is precisely the kind of information left unsaid. Who would write, “If I can walk inside my house, I know that my house is bigger than I am?” Nevertheless, we naturally understand that: $x\textit{ walk-inside }y\implies x<^{\text{size}}y$.

Directly attacking the link between affordances and properties requires access to implications across the edges. Without such information, we can use objects as a proxy to understand how much modern neural networks know about this edge. For example, taking an object like boots, and using only its top affordances wear, kick off, and lace up, can we predict its properties?

Experiments

As shown at the bottom of Figure 1, our problem space naturally defines three edges in a graph. A property prediction task may attempt to produce the human-labeled set of properties given a new object (O$\rightarrow$P) (?, ?). Predicting affordances can be done similarly: given a new object, can its top affordances be distinguished from others (O$\rightarrow$A)? And finally, the troublesome but fertile edge between properties and affordances: can a model predict the set of properties compatible with an affordance (A$\rightarrow$P)?

We frame each scenario as a series of joint reasoning tasks. Given two instances (e.g., an object and a property), a model must make a binary decision as to whether they are compatible. For example, predicting which of $k$ properties $\{p_{1},\ldots,p_{k}\}$ are compatible with an object $o$ will be set up as $k$ compatibility tasks $(o,p_{i})\rightarrow\{0,1\}$.We experimented with other task setups found in previous work, such as using an object to predict a $k$-length vector of properties: $(o)\rightarrow\{0,1\}_{k}$. However, we found models performed better on all metrics by instead framing the task as a series of compatibility decisions. We suspect the reason is that this setup allows models to take advantage of input representations of both words rather than just one. We denote the tasks as object-property (O$\longleftrightarrow$P), object-affordance (O$\longleftrightarrow$A), and affordance-property (A$\longleftrightarrow$P).

Data

To fuel experiments in these three tasks, we introduce two new datasets. The first we call the abstract dataset, which is a set of judgements elicited from only the name of the object (e.g., wheelbarrow) and property (e.g., is an animal). The second is the situated dataset, where properties and affordances are annotated on objects in the context of real-world pictures.Annotations for both datasets are performed by workers on Amazon Mechanical Turk.

Several lists of properties (?, ?), categorization schemes (?, ?), and quantification layers (?, ?) have been proposed. We take the set of objects and properties from ? (?) and perform filtering and preprocessing similar to ? (?). We also include the set of objects from the MS COCO dataset (?, ?), collapse similar objects (e.g., many bird species) and add seven new properties (such as man-made and squishy). We end up with a set of 514 objects and 50 properties. We re-annotate all 25,700 object-property pairs to eliminate false negatives from the original McRae data collection process and provide labels for new entries. We annotate each pair three times for a total of 77,100 annotations, and keep only labels with $\geq 2/3$ agreement.

Situated Dataset

We also annotate instances of objects situated in photographs. Images have the great advantage of resolving visual ambiguities of appearance, shape, and form. For example, a bottle has different properties if it is a glass beverage container or plastic shampoo tube. Only a few non-visual properties (e.g., smelliness) must then be inferred from the environment.

To build the an experimental situated testbed, we sample images from the MS COCO dataset (?, ?). We constrain each image to have between three and seven objects to avoid scenes that are too sparse (often portraits) or dense (cluttered collections). We also ensure that we have at least five samples of each of the 80 unique object categories in the dataset. We end up with 1,024 objects across 220 images. We then annotate all 50 properties (introduced in the abstract dataset) for each object, annotating each three times for a total of 153,600 labels. We filter using the same scheme ($\geq 2/3$ agreement).

In addition to the properties, we also collect annotations of the affordances for all objects in the situated dataset. We allow annotators to choose from the 504 verbs from the imSitu dataset (?, ?). We provide common variants of each verb that include particles, allowing annotations such as pick up and throw out. Annotators select the top three to five affordances that come to mind when they see the selected object in the context of its photograph. We again perform this annotation three times for each object, and aggregate the verbs chosen to pick the top three most common affordances for each object. We end up with a set of sparsely labeled affordances for each situated object. We perform balanced negative sampling by selecting $k=3$ affordances for each datum and setting their labels to zero.

Detailed statistics and examples for both datasets are shown in Table 1. Full lists of the objects, properties, and affordances, as well as the annotation interfaces, are provided in the Appendix.

Models

Contextualized representations

The other two representations are ELMo (?, ?) and BERT (?, ?), which are contextualized. These require full sentences (as opposed to single words) to compute a vector, but in turn produce results more specific to words’ linguistic surroundings. For example, ELMo and BERT produce different representations for book in “I read the book” versus “Please book the flight,” while word embeddings have only a single representation.

To account for this, we generate sentences using the relevant objects, properties, and affordances for the task at hand. For example, to judge accordion and squishy, we would generate “An accordion is squishy.”

For ELMo, we then take the final layer representations for the two compared words, each of which is a $d=1024$ length vector. For BERT, we take the overall sentence representation as the final layer’s hidden state of the [CLS] (sentence summary) symbol, which produces a single $d=1024$ vector.

Finetuning

Given the word representations above, we finetune each of the models by adding trainable multilayer perceptron (MLP) after the input representations. This allows models to learn interrelations between the two categories at hand, essentially calibrating the unsupervised representations into a compatibility function. We use a single hidden layer in the MLP, and train using mean squared error loss with L2 regularization. For BERT, we find the standard procedure of finetuning the entire model vital for good performance.

To summarize, for two words $(w_{i},w_{j})$ which can be written together in a sentence $s=w_{1}...w_{n}$, we have for a model $m$,

We optimize models using gradient descent (or Adam (?, ?) for BERT), and tune all hyperparameters using $k$-fold cross validation with $k=5$.

Baselines

We compare performance for these models against two simple approaches. The random baseline simply flips a coin for each compatibility decision. The majority baseline uses the per-class majority label for the training set, aggregating by property for the $O\longleftrightarrow P$ and $A\longleftrightarrow P$ tasks, and by affordance for the $O\longleftrightarrow A$ task.

Human performance

Finally, we estimate human performance on this task. We sample 50 samples at random from the test set for each task, and have an expert annotate them. For fairness to the models, we do not show the expert the photographs or exact instance from which the situated examples are drawn.

Results

A summary of all model performances is shown in Table 2. Consistent with prior work that has studied object and property compatibility (?, ?), we find good but not perfect performance (close to 0.70 F1 scores) on the abstract dataset (task $O\longleftrightarrow P$). Models fare slightly worse on the situated $O\longleftrightarrow P$ task, with the best performance below 0.60 F1. This effect is consistent in the human expert scores as well. Though this dataset is larger, the introduction of context allows for greater variance in the properties of an object.

The object-affordance compatibility task ($O\longleftrightarrow A$) yields significantly higher numbers. Not only is this task statistically easier (as demonstrated by the strong majority baseline), but this edge is the only one directly observed in language. All models pretrained on text have been exposed to many instances of likely verbs for each object considered. In fact, all pretrained models perform in the same range as human ability, and there is no statistically significant difference between the models for this task.

However, all models struggle with the affordance-property task ($A\longleftrightarrow P$). The highest F1 scores are in the 0.30s, with the random baseline achieving the highest macro F1 score by property. While this task is also the most difficult for humans, their macro F1 scores for both affordances and properties are around double those of the best performing models. We posit that the inference between affordances and properties requires multi-hop reasoning that is simply not present in the pretraining of large text-based models. We provide further analysis in the following section.

Analysis

Models achieve reasonable performance predicting the compatibility of both properties and affordances with objects. However, the task requiring inference between affordances and properties (A$\longleftrightarrow$P) confounds even the strongest models.

We explore this result through a detailed analysis of the top performing model. Figure 2 presents a breakdown of BERT’s results on the affordances-property compatibility task (A$\longleftrightarrow$P) on the situated dataset. From the leftmost graph (a), we observe that a per-property analysis shows a largely bimodal split between properties that are fully predicted (1.0 F1), and went completely unmodeled (0.0 F1). Affordances, on the other hand, lie more evenly across the F1 range. Because the task involved the compatibility between properties and affordances, mass for correct predictions must be shared between the two data groups. That so few properties achieved a high F1 score suggests that many affordances rely on only a few properties for accurate prediction.

We perform further analysis to investigate which kinds of properties yielded better affordance-property modeling. We categorize each property into four coarse classes: functional (e.g., is used for cooking), encyclopedic (e.g., is an animal), commonsense (e.g., comes in pairs), and perceptual (e.g., is smooth). Figure 2 (b) shows a breakdown of property performance grouped by these four categories. (Here, we plot accuracy instead of the sharper F1 metric to better illustrate the spread of performance.) Functional properties exhibit the highest performance. This makes intuitive since, because functional capabilities are directly tied to an object’s affordances. In contrast, perceptual properties exhibit generally lower and inconsistent performance than other categories. We suspect that perceptual observations observed in text are not expressed with affordances, making this connection difficult for models. Largely perceptual features can be written about with simple verbs (hear, see, feel), giving them less implicit evidence than more nuanced properties. Finally, encyclopedic and commonsense properties fall somewhere in the middle. These properties, which involve an object’s general characteristics (like requires gasoline, lives in water, or has a peel), correlate with a variety of verbs. But they may only be directly expressed at a distance from a verb, making the inference between them still challenging.

Our final analyses in Figure 2 (c) and (d) investigate whether there is a link between the predictive power of the model and how often a word is used in text. We compute the frequencies of all affordances and properties occurring in natural language using the Google Web 1T corpus, an n-gram corpus computed from approximately one trillion words (?, ?). Figure 2 (c) plots the F1 score of properties against how frequently they appear in natural language; 2 (d) plots the same for affordances. We include a best-fit line along with confidence intervals shown as one standard deviation of the data. We do not observe a statistical correlation between how much affordances and properties are written about, and how well neural models are able to connect their effects; a single confidence interval spans both positive and negative slopes. This lack of clear correlation is surprising, because large state-of-the-art neural textual models generally improve with repeated exposure to instances of words. Except for the three most common words measured by property F1 score, the rest of the data shows a strikingly uniform distribution of F1 scores for any choice of frequency in natural language. This suggests that current neural models are fundamentally limited in their capacity for physical reasoning, and that only new designs—not more data—can allow them to acquire this skill.

Discussion

Despite being able to associate a considerable range of information with the names of objects, neural models are not able to capture the more subtle interplay between affordances and properties. In some sense, this result is unsurprising. Collecting information around an object can be informed largely by the co-occurrence of words around that object’s various mentions. Affordances that imply properties (and the reverse) are rarely mentioned together; their mutual connotation naturally renders joint expression redundant. Hence, priorless models that learn from statistical associations falter. Given the depth of the networks used in models such as ELMo and BERT, complex inter-parameter structure arises, but the latent semantic patterns that describe physical commonsense are much weaker than more superficial patterns that arise due to grammar or domain.

This evidence evokes theories of embodied cognition (?, ?, ?), which suggest that the nature of human cognition depends strongly on the stimuli granted by physical experience. If this is so, then how is information encoded in our physical experience such that we can make predictions? If we assume a form of mental simulation, then what are the mental limits on its reliability? From an artificial intelligence perspective, the more interesting proof is in the principles of creating such a mental simulator. If we are to simulate human capacity for thought, how actually must we simulate elements of the physical world?

With the rise of physics engines, our ability to model physical inferences grows (?, ?). However, while this may make us better at anticipating human predictions about physical situations through perceptual stimuli (?, ?), there is still a long way to go before we understand the inferences that are being made through more symbolic stimuli, such as language. Exploring the mechanisms underlying this communication using an implicit shared world model will require us to either develop access to such a world model, or expose algorithms to predictions of that world model by directly querying humans. Bridging the inductive biases learned from simulation (?, ?) and those discovered by scientists (?, ?) to make inferences implicit in text will lead to a more cohesive model of commonsense physics. We expect such a model to bear fruit in studies of communication rich with physical implications.

Acknowledgments

This work was supported by NSF grants (IIS-1524371, 1637479, 1703166), NSF Fellowship, the DARPA CwC program through ARO (W911NF-15-1-0543), and gifts by Google and Facebook. The views and conclusions contained herein are those of the authors and should not be interpreted as representing endorsements of the funding agencies.

References

Appendix

We provide below a full list of the objects considered in both of our datasets. We note the split that it belongs to in each dataset (train or test), and the origin of the object (MR = ? (?); C = MS COCO (?, ?)).

In general, this list is the union of objects found in McRae (?) and MS COCO. For cases where we do not use the object in either dataset (i.e., a “-” in both columns), we mark the row in italics and provide a note for why it was dropped. The most common reasons for dropping an object are: polysemy, such as in bat (animal vs baseball); hypernomy (e.g., sparrow $\rightarrow$ bird) to collapse similar objects, such as the eighteen species of birds found in McRae (?); and dialect (e.g., trousers $\rightarrow$ pants) to make annotations easier for readers of American English. (For the several species of fish in McRae (?), we left two fish of clearly distinct sizes, goldfish and trout, rather than including a generic “fish” object.)

We provide below the list of candidate verbs, along with the assistive particles and prepositions, we used for annotating affordances.

When annotating “What might you do to the X?” for an object X, it can be helpful or necessary to use a particle or preposition when writing an answer. Here are some examples:

We use two strategies to enable annotators to write grammatical constructions like the above. For particles, we provide common variants of each verb that use particles, such as buckle up and buckle in in addition to buckle. For prepositions, we allow an additional choice of a preposition after the annotator has selected a verb (or verb + particle) from the list.

We discard both particles and prepositions when building our task data. We do this for two reasons. First, we wish to eliminate errors where an annotator mistakes the subtle distinction between a particle and a preposition. Second, we want to constrain the input and output space for the models, which would otherwise multiplicatively scale the verbs by the number of particles and prepositions.

To pick the verbs, we take the set of 504 verbs used in the imSitu dataset (?, ?) and lemmatize them. For particles and prepositions, we run a dependency parser on a large corpus of sentences and aggregate statistics. We add all verb + particle forms of all verbs that occur in at least 5% of the usages of that verb. We then provide the twenty-six widely used prepositions as an additional selection. For brevity, we list here just the verbs in their lemmatized form, and provide the twenty-seven unique particles and prepositions.

adjust, admire, ail, aim, applaud, apply, apprehend, arch, arrange, arrest, ascend, ask, assemble, attach, attack, autograph, bake, balloon, bandage, baptize, barbecue, bathe, beg, bet, bike, bite, block, blossom, board, boat, bother, bounce, bow, braid, branch, brawl, break, brew, browse, brush, bubble, buckle, build, bulldoze, burn, bury, butt, butter, button, buy, call, calm, camouflage, camp, caress, carry, cart, carve, catch, celebrate, chase, check, cheer, cheerlead, chew, chisel, chop, circle, clap, claw, clean, clear, clench, climb, cling, clip, coach, collide, color, comb, communicate, commute, compete, complain, confront, congregate, construct, cook, cough, count, cover, craft, cram, crash, crawl, crest, crouch, crown, crush, cry, curl, curtsy, dance, decompose, decorate, deflect, descend, destroy, detain, dial, din, dip, discipline, discuss, disembark, display, dissect, distract, distribute, dive, dock, douse, drag, draw, drench, drink, drip, drive, drool, drop, drum, dry, duck, dust, dye, eat, educate, eject, embrace, emerge, empty, encourage, erase, erupt, examine, exercise, exterminate, extinguish, fall, farm, fasten, feed, fetch, fill, film, fish, fix, flame, flap, flex, flick, fling, flip, float, floss, fold, forage, ford, frisk, frown, fry, fuel, gamble, garden, gasp, gather, giggle, give, glare, glow, glue, gnaw, grieve, grill, grimace, grin, grind, guard, handcuff, hang, harvest, haul, heave, help, hike, hit, hitchhike, hoe, hoist, hug, hunch, hunt, hurl, ignite, ignore, imitate, immerse, inflate, inject, insert, instal, instruct, intermingle, interrogate, interview, jog, juggle, jump, kick, kiss, knead, kneel, knock, lace, land, lap, lather, laugh, launch, lead, leak, lean, leap, lecture, lick, lift, light, load, lock, make, manicure, march, mash, massage, measure, mend, microwave, milk, mime, mine, misbehave, moisten, moisturize, mold, mop, mourn, mow, nag, nail, nip, nuzzle, offer, officiate, open, operate, overflow, pack, package, paint, panhandle, parachute, parade, paste, pat, paw, pay, pedal, pee, peel, perform, perspire, phone, photograph, pick, pilot, pin, pinch, pitch, place, plant, plow, plummet, plunge, poke, poop, pot, pounce, pour, pout, practice, pray, preach, press, prick, protest, provide, prowl, prune, pry, pucker, pull, pump, punch, punt, push, put, queue, race, raft, rain, rake, ram, read, rear, reassure, record, recover, recuperate, rehabilitate, release, repair, rest, restrain, retrieve, rid, rinse, rock, rot, row, rub, run, salute, say, scold, scoop, score, scrap, scratch, scrub, seal, sell, serve, sew, shake, sharpen, shave, shear, shell, shelve, shiver, shoot, shop, shout, shovel, shred, shrug, shush, sign, signal, sing, sit, skate, sketch, ski, skid, skip, slap, sleep, slice, slide, slip, slither, slouch, smash, smear, smell, smile, sneeze, sniff, snow, snuggle, soak, soar, socialize, sow, spank, speak, spear, spill, spin, spit, splash, spoil, spray, spread, sprinkle, sprint, sprout, spy, squeeze, squint, stack, stampede, stand, staple, star, steer, sting, stir, stitch, stoop, storm, strap, stretch, strike, strip, stroke, study, stuff, stumble, subdue, submerge, suck, surf, swarm, sweep, swim, swing, swoop, tackle, talk, tap, taste, tattoo, taxi, teach, tear, telephone, throw, tickle, tie, till, tilt, tip, tow, train, trim, trip, tug, tune, turn, twirl, twist, type, uncork, unload, unlock, unpack, unplug, unveil, urinate, vacuum, vault, videotape, vote, wad, waddle, wag, wait, walk, wash, water, wave, wax, weed, weep, weigh, weld, wet, wheel, whip, whirl, whisk, whistle, wilt, wink, wipe, work, wrap, wring, wrinkle, write, yank, yawn

Particles and Prepositions

about, after, against, around, as, at, before, behind, by, down, for, from, in, into, like, of, off, on, onto, out, over, through, to, towards, up, with, without

We provide below the data collection interfaces we used to label properties in both the abstract and situated datasets. The affordances collected for the situated dataset used a similar interface.

The interface for labeling the abstract dataset asks about an object by giving only its name (e.g., accordion), and then asking about the usual properties of that object. Annotators are given the choice “too difficult to tell,” but are encouraged to use that only when absolutely necessary, and use their best guess when possible. We display twenty-five properties at once.

Situated Dataset

The interface for labeling the situated dataset displays a picture with an object highlighted in it (the photos and object labels are from MS COCO (?, ?)).

To label properties, the annotator is asked to select which properties apply to this particular object. Because the object is grounded in a specific instance, we remove the “too difficult to tell” option, forcing a yes/no decision. We display ten properties at once.

To label affordances (not pictured), the annotator is prompted “What might you do to X?”, where X is the highlighted object. The annotator is asked to provide three to five choices using the provided verbs, particles, and prepositions (described above). After collecting all annotations, we discard the particles and prepositions, and aggregate to pick the top three verbs used for each instance. To create negative samples, we randomly pick three verbs that were not selected from the complete list.

Quality Control

We collect annotations using Amazon Mechanical Turk. We find workers generally provide high quality annotations. However, even with strict qualification requirements, we often find a nontrivial rate of negligence. To combat this, we inject two pseudo-properties (e.g., “Is X a word in the English language?” for the object X) in our data collection interface. Because we know the answers to these questions in advance, we can use them as a check to prevent a worker from answering at random. We discard all data from any worker who answers any of these questions incorrectly.

This is the first version uploaded to arXiv. Previously, BERT was trained in the same way as all other models: fixing the model, and training an MLP on top. We fine-tuned BERT end-to-end, which then outperformed all other models. We updated the Models section, results and statistical significance tests (Table 2), and the analysis graphs (Figure 2). While BERT’s finetuned numbers are higher overall, the conclusions of the paper remain unchanged. BERT’s performance on the situated affordance $\longleftrightarrow$ property task are still far below humans’.

This version also includes an Appendix, which contains detailed information about our datasets.