Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision

Introduction

Most humans learn language understanding from multiple modalities rather than only from the text and audio, especially using the visual modality. As claimed in Bloom (2002), visual pointing is an essential step for most children to learn meanings of words. However, existing language pre-training frameworks are driven by contextual learning which only takes the language context as self-supervision. For example, word2vec Mikolov et al. (2013) takes surrounding bag-of-words; ELMo Peters et al. (2018) and GPT Radford et al. (2018) take succeeding contexts; and BERT Devlin et al. (2019) takes randomly masked tokens. Although these self-supervised frameworks have achieved strong progress towards understanding human language, they did not borrow grounding information from the external visual world (see related motivations in recent work by Bender and Koller (2020) and Bisk et al. (2020)).

In this paper, we introduce the visually-supervised language model that simulates human language learning with visual pointing Bloom (2002). As shown in Fig. 1, this model takes language tokens as input and uses token-related images as visual supervision. We name these images as vokens (i.e., visualized tokens), since they act as visualizations of the corresponding tokens. Assuming that a large aligned token-voken dataset exists, the model could learn from these vokens via voken-prediction tasks.

Unfortunately, such an aligned token-voken dataset is currently unavailable and hence there are two main challenges in creating it from visually-grounded language datasets. First, there is a large discrepancy between visually-grounded language (which provides innate visual grounding supervision) and other types of natural language. For example, about 120M tokens are available in visually-grounded language datasets Tan and Bansal (2019); Chen et al. (2019), which is far less compared to the 3,300M tokens in BERT training data and 220B tokens in T5 Raffel et al. (2019). Grounded language also prefers short and instructive descriptions, and thus has different distributions of sentence lengths and active words to other language types. Second, most of the words in natural language are not visually grounded, hence this challenges the premise in creating visual supervision. With an approximate estimation, the ratio of grounded tokens is only about $28\%$ in English Wikipedia. This low grounding ratio leads to low coverage of visual supervision in previous approaches Frome et al. (2013); Kiela et al. (2018).

To resolve the above two challenges, we propose our vokenization method (as shown in Fig. 1) that contextually maps the tokens to the visualized tokens (i.e., vokens) by retrieval. Instead of directly supervising the language model with visually grounded language datasets (e.g., MS COCO Lin et al. (2014)), we use these relative small datasets to train the vokenization processor (i.e., the vokenizer). We then generate vokens for large language corpora (e.g., English Wikipedia), and our visually-supervised language model will take the input supervision from these large datasets, thus bridging the gap between different data sources, which solves the first challenge. The second challenge of low grounding ratio seems to be an inherent characteristic of language; however, we observe that some non-visually-grounded tokens can be effectively mapped to related images when considering its context, e.g., the abstract word “angry” in the sentence “an angry cat lies on my leg”. This observation is realized by our contextual token-image matching model (defined in Sec. 3.2) inside our vokenization processor, where we map tokens to images by viewing the sentence as the context.

Using our proposed vokenizer with a contextualized token-image matching model, we generate vokens for English Wikipedia. Supervised by these generated vokens, we show consistent improvements upon a BERT model on several diverse NLP tasks such as GLUE Wang et al. (2019), SQuAD Rajpurkar et al. (2016), and SWAG Zellers et al. (2018). We also show the transferability of our vokens to other frameworks (i.e., RoBERTa).

Visually-Supervised Language Models

Contextual language representation learning is driven by self-supervision without considering explicit connections (grounding) to the external world. In this section, we illustrate the idea of a visually-supervised language model and discuss the challenges of creating its visual supervision.

To provide visual supervision to the language model, we assume a text corpus where each token is aligned with a related image (although these voken annotations currently do not exist, we will try to generate vokens next in Sec. 3 by the vokenization process). Hence, these images could be considered as visualizations of tokens and we name them as ‘vokens’. Based on these vokens, we propose a new pre-training task for language: voken classification.

2 The Voken-Classification Task

This task could be easily integrated into current language pre-training frameworks, and we next show an example.

Fig. 2 shows an example realization of the voken-classification task that provides visual supervision to BERT Devlin et al. (2019). The original BERT pre-training mainly relies on the task of masked language modelThe next-sentence prediction task is removed in RoBERTa Liu et al. (2019) and XLM Lample and Conneau (2019) and the fine-tuning results are not largely affected. (illustrated on the left side of Fig. 2): tokens are randomly masked and the model needs to predict these missing tokens from language context. For simplicity, we use $s$ and $\hat{s}$ to denote the set of tokens and masked tokens, separately. The unmasked tokens are the set difference $s\setminus\hat{s}$. Suppose $q_{i}$ is the conditional probability distribution of the $i$-th token, the Masked Language Model (MLM) loss is the negative log-likelihood of the masked tokens:

Without changing the model and model’s inputs, we calculate the voken-classification loss for all tokens (illustrated on the right side of Fig. 2):

The visually-supervised masked language model takes the sum of these two losses with a ratio $\lambda$.

3 Two Challenges in Creating Vokens

Previous sections illustrate the potential external supervision by assuming the existence of vokens. However, we are currently lacking the dense annotations from tokens to images. The most similar concept to vokens is phrase localization (e.g., in Flickr30K entities Young et al. (2014); Plummer et al. (2017)). Because the process of collecting phrase localization is costly, the coverage and the amount of annotations cannot meet our requirements.Recently, a concurrent work Pont-Tuset et al. (2019) releases localized narratives. The tokens are aligned with image pixels instead of images. Apart from phrase localization, the most promising data source is image captioning datasets with sentence-to-image mappings (or discovered from multimodal documents, as in Hessel et al. (2019)). Image captions belong to a specific type of language called grounded language Roy and Pentland (2002); Hermann et al. (2017), which has an explicit grounding to external existence or physical actions. However, grounded language has a large discrepancy to other types of natural language (e.g., News, Wiki, and Textbooks). To illustrate this, we list key statistics of three image-captioning dataset (i.e., MS COCO Lin et al. (2014), Visual Genome Krishna et al. (2017), and Conceptual Captions Sharma et al. (2018)) and three language corpora of other language types (i.e., Wiki103 Merity et al. (2017), English Wiki, and CNN/Daily Mail See et al. (2017)) in Table 1. This discrepancy between grounded language and other types of natural language leads to two challenges:

A. Different Distributions between Grounded Language and Other Natural Language Corpora. Sentences belonging to grounded language are usually short and informative, e.g., the average sentence length in MS COCO is $11.8$, which is much shorter than the average sentence length of $24.1$ in English Wiki. The vocabularyThe vocabulary is calculated following Karpathy and Fei-Fei (2015) where the words with $>5$ occurrence is counted. of MS COCO only covers around one-third of token types Smith (2019) in English Wiki. There is also a large divergence of the 1-Gram and 2-Gram distributions (measured by Jensen–Shannon divergence) between grounded language dataset and the English Wikipedia. Lastly, the amount of tokens in grounded language corpora are also orders of magnitude smaller than commonly-used Wikipedia.

B. Low Grounding Ratio in Natural Language. The grounding ratio is defined as the percentage of visually grounded tokens in the dataset. Visually grounded tokens (e.g., concrete nouns) are the token types that are naturally related to specific visual contents (e.g., ‘cat’, ‘cake’, ‘clock’). Since a precise list of such token types is hard to define, we thus estimate the grounding ratio based on existing grounded language corpora. Specifically, we consider a token type with more than $100$ occurrences in MS COCO (after removing all stop words) as visually-grounded. A sample of these token types could be found in the Appendix. As shown in the last column of Table 1, the grounding ratio of English Wiki is $27.7\%$, which is almost half of that in Visual Genome.

To address these two challenges, we propose a vokenizer with contextual token-image matching models next in Sec. 3.

Vokenization

In the previous section, we discuss the potential of using vokens (i.e., visualized tokens) as visual supervision to the language model, and also demonstrate the large gap between currently available resources (i.e., annotated dataset) and the desired requirements. Hence, in this section, we develop a framework that can generate vokens. As shown in Fig. 2, the general idea is that we learn a “vokenizer” from image-captioning dataset and use it to annotate large language corpora (i.e., English Wiki), thus bridging the gap between grounded language and other types of natural language. We start by illustrating the vokenization process and then describe how we implement it.

2 Contextual Token-Image Matching Model

Lying in the core of the vokenization process is a contextual token-image matching model. The model takes a sentence $s$ and an image $x$ as input, and the sentence $s$ is composed of a sequence of tokens $\{w_{1},w_{2},\ldots,w_{l}\}$. The output $r_{\theta}(w_{i},x;s)$ is the relevance score between the token $w_{i}\in s$ and the image $x$ while considering the whole sentence $s$ as a context.

To model the relevance score function $r_{\theta}(w_{i},x;s)$, we factorize it as an inner product of the language feature representation $\bm{f}_{\theta}(w_{i};s)$ and the visual feature representation $\bm{g}_{\theta}(x)$:

These two feature representations are generated by language and visual encoders respectively. The language encoder first uses a pre-trained $\text{BERT}_{\text{BASE}}$ Devlin et al. (2019) model to contextually embed the discrete tokens $\{w_{i}\}$ into hidden-output vectors $\{\bm{h}_{i}\}$:

Then we apply a multi-layer perceptron (MLP) $\mathit{w\mbox{\_}mlp}_{\theta}$ to down project the hidden output $\bm{h}_{i}$. In order to simplify the retrieval process in Sec. 3.1, the final language features are normalized to norm-1 vectors by dividing their Euclidean norms:

On the other side, the visual encoder first extracts the visual embedding $e$ from a pre-trained ResNeXt Xie et al. (2017). Similar to the language encoder, an MLP layer $\mathit{x\mbox{\_}mlp}_{\theta}$ and an L2-normalization layer are applied subsequently:

Training

Since the dense annotations from tokens to images are lacking and hard to generate (illustrated in Sec. 2.3), we thus alternatively train the token-image matching model from weak supervision in image-captioning datasets (e.g., MS COCO Lin et al. (2014)). These datasets are comprised of sentence-image pairs $\{(s_{k},x_{k})\}$ where the sentence $s_{k}$ describes the visual content in image $x_{k}$. To build alignments between tokens and images, we pair all tokens in a sentence $s_{k}$ with the image $x_{k}$. The model is then optimized by maximizing the relevance score of these aligned token-image pairs over unaligned pairs.

Without loss of generality, assuming $(s,x)$ is an image-captioning data point, we randomly sample another image $x^{\prime}$ with the condition $x^{\prime}\neq x$. We then use hinge loss to optimize the weight $\theta$ so that the score of the positive token-image pair $r_{\theta}(w_{i},x;s)$ aims to be larger than the negative pair $r_{\theta}(w_{i},x^{\prime};s)$ by at least a margin $M$.

Intuitively, minimizing this hinge loss $\max\{0,$ $M-\mathit{pos}+\mathit{neg}\}$ will try to increase the score of the positive pair and decrease the score of the negative pair when the score difference is smaller than the margin $M$. Otherwise (if the difference is $\geq$ margin $M$), the two scores remain unchanged.

Inference

Given that the relevance score is factorized as the inner product of feature representations $\bm{f}_{\theta}(w_{i};s)$ and $\bm{g}_{\theta}(v)$, the retrieval problem in Sec. 3.1 could be formulated as Maximum Inner Product Search Mussmann and Ermon (2016)). Moreover, since the vectors are norm-1, the vector with the maximum inner product is identical to the closest vector in the Euclidean space (i.e., Nearest Neighbor Knuth (1973)). We illustrate the detailed implementation in Fig. 3.

3 Revokenization

A constraint of the vokenization process in Sec. 3.1 is that the vokens depend on the actual tokenizer of the language encoder in Sec. 3.2. Since different frameworks utilize a various range of tokenizers, this constraint limits the transferability of vokens between different frameworks. Instead of binding our vokenizer to a specific pre-training framework (e.g., BERT), we want to enable its extensibility to other frameworks (e.g., RoBERTa). Thus, we introduce a “revokenization” technique to address this limitation.

Given two different tokenizers $T_{1}$ and $T_{2}$, they tokenize a sentence $s$ into two different sequences of tokens: $T_{1}(s)=\left(w_{1},w_{2},\ldots,w_{l}\right)$ and $T_{2}(s)=\left(u_{1},u_{2},\ldots,u_{m}\right)$. Without loss of generality, assuming the vokenizer is built based on the first tokenizer $T_{1}$, the standard vokenization process will generate a sequence of vokens $\{v(w_{i};s)\}_{i=1}^{l}$ which are one-to-one aligned with the tokens $\{w_{i}\}_{i=1}^{l}$. Our goal is to transfer these $w$-related vokens to the $u$-related vokens generated by $T_{2}$. We adapt the idea of “nearest neighbor algorithm” Altman (1992) here. For a given token $u_{j}$, among all $w$’s, we select the one that overlaps the most with $u_{j}$ and record it as $w_{\mathit{ind}(j)}$. The voken for $u_{j}$ is defined as the voken for its “nearest neighbor” $w_{\mathit{ind}(j)}$:

Experimental Setups and Results

We train our model on English Wikipedia BERT Devlin et al. (2019) also uses Toronto Books Corpus Zhu et al. (2015). However, the dataset is not publicly released. We thus exclude it in our study to ensure reproducibility. and its featured subset Wiki103 Merity et al. (2017). We use our vokenizer to generate vokens for these two datasets as well. The pre-trained models are then fine-tuned on GLUE Wang et al. (2019), SQuAD Rajpurkar et al. (2016, 2018), and SWAG Zellers et al. (2018) to assess the pre-training performance. Since some smaller tasks in GLUE are reported as unstable Dodge et al. (2020), recent papers (e.g., Li et al. (2020b)) only report on selected tasks. We follow this trend and evaluate on the four largest datasets (i.e., SST-2 Socher et al. (2013), QNLI Rajpurkar et al. (2016), QQP Iyer et al. (2017), MNLI Williams et al. (2018)).The size of the used four dataset range from $60$K to $400$ while the omitted dataset range from $0.6$K to $8.5$K..

2 Implementation Details

When pre-training the model on pure language corpus, we unify the training protocols to avoid possible side effects. We follow previous works to conduct two simplifications: 1. Removing the next-sentence-prediction task Liu et al. (2019) 2. Using fixed sequence length Conneau et al. (2020) of $128$. We take the $12$-layer $\text{BERT}_{\text{BASE}}$ model of $768$ hidden dimensions and train it on English Wikipedia for $200$K steps from scratch. We also take a reduced $6$-layer model and train it on Wiki103 for $40$ epochs ($160$K steps) because this reduced model could not fit the full English Wikipedia dataset.

Since we only use the vokens in the supervision, the voken-classification task does not bring additional parameters to the language model but needs more computations. We thus adjust the training steps for pure masked-language-model (MLM) training accordingly for a fair comparison. The loss ratio $\lambda\mbox{=}1.0$ in Eqn. 1 is not tuned because of limited budget. All pre-training processes take batch sizes of $256$ and learning rates of $2e\mbox{-}4$. For fine-tuning tasks, we report the results on the validation sets. We train $3$ epochs with a learning rate of $1e\mbox{-}4$ and a batch-size of $32$ for all tasks in GLUE. The hyper-parameters for SQuAD, SWAG are borrowed from BERT.

3 Results

As reported in Table 2, we fine-tune the pre-trained models on different natural-language tasks. The models are either pre-trained with masked language model (e.g., “$\text{BERT}_{\text{6L/512H}}$”) or pre-trained with masked language model with an additional voken-classification task (e.g., “$\text{BERT}_{\text{6L/512H}}$+Voken-cls”) following Eqn. 1. The default metric is accuracy. Following Wang et al. (2019), we report the average of F1 and accuracy for QQP. For SQuAD, we report the exact matching and F1 score respectively. We also compute macro-averages for evaluated tasks (denoted as “Avg.” in the last column) as a general indicator. Although the different architectures of models (i.e., 6L/512H and 12L/768H) affect the fine-tuning results, the voken-classification task consistently improves the downstream tasks’ performance and achieves large average gains. We also show the transferability of our vokenizer to the RoBERTa model and observe the same phenomenon as that in BERT.

Analysis

In Sec. 2.3, we illustrated the differences between (visually-)grounded-language datasets and other natural-language corpora by demonstrating their contrasting statistics. In this section, we study the models trained with grounded language and show their ineffectiveness on pure-language tasks. We first investigate vision-and-language pre-training frameworks, which succeed on multimodal tasks. As shown in Table 3, when fine-tuning them on pure-language tasks, the results are generally lower than the pre-trained BERT model.ViLBERT Lu et al. (2019) freezes the BERT weight in its training thus their results are the same to BERT; Uniter Chen et al. (2019) shrinks its vocab thus is not shown. Although these frameworks are different in multiple ways, the only remarkable factor to the fine-tuning results is the BERT-weight initialization. Moreover, we also show that these models are similar to a BERT model with a random weight noise of the same magnitude. We thus claim that vision-and-language pre-training on visually-grounded language dataset currently might not help the pure-language tasks. Note that the BERT results in Table 2 are not fairly comparable to the results in Table 3 because the original BERT model Devlin et al. (2019) also uses Toronto Books Corpus Zhu et al. (2015). Unfortunately, this dataset is not publicly available and hence we exclude it. According to Raffel et al. (2019), the exclusion of Toronto Books Corpus downgrades the results and we observe the same tendency here (comparing $\text{BERT}_{\text{12L/768H}}$ in Table 2 and $\text{BERT}_{\text{BASE}}$ in Table 3).

Besides these existing models, we next investigate the BERT models trained with masked language model on grounded language data (i.e., MS COCO). A control experiment is built by shrinking the Wiki103 to the same token amount as MS COCO. We also provide the BERT model trained from scratch as a baseline. As shown in Table 4, the model trained with MS COCO is significantly worse than the model trained with Wiki103 on all downstream tasks. The reason might be the large discrepancy between visually-grounded language and other types of language as shown in Sec. 2.3.

2 Token-Level vs. Sentence-Level Approaches

In Sec. 1, we stated the drawbacks of the purely sentence-level and token-level approaches, then introduce the contextual token-level approach (i.e., the contextual token-image matching model in Sec. 3.2) which combines these two approaches. In this section, we demonstrate a careful comparison between our vokenization process and the other two approaches from two perspectives: the retrieval methods and the supervision types. Experiments are conducted with the same hyper-parameters and dataset as “$\text{BERT}_{\text{12L/768H}}$+Voken-cls” in Table 2.

To conduct sentence-level retrieval, we first adapt the contextual token-image matching model in Sec. 3.2 to a sentence-image matching model (details in Appendix). We then retrieve a related image for each sentence. As shown in Table 5, these retrieved images are used as two kinds of supervisions by putting classifiers at different places: in the row “SentLabel”, we provide sentence-level supervision by using the classifier to predict the label for the whole sentence (similar to the BERT’s “next-sentence prediction” (NSP) task); and in the row “Propagated”, we provide token-level supervision by propagating sentence-level labels to all tokens in the sentences, and apply the classifier at each token (similar to our voken-classification task). The results of both kinds of supervisions are lower than our proposed vokens (in the row “Vokens”). One possible reason for these lower results is that finding an image that conveys the meaning of the whole sentence is hard. We also find that dense token-level supervision also outperforms the sentence-level supervision.

Token-level Retrieval

Our proposed vokenization process is viewed as contextual token-level retrieval, which grounds tokens with whole sentences as context. We here consider a purely token-level retrieval method regarding term frequencies. The term frequency $\mathit{tf}(\mathit{tok},x_{i})$ Manning et al. (2008) is calculated based on the occurrence $\#(\mathit{tok},x_{i})$ of the token $\mathit{tok}$ in the image $x_{i}$’s captions.

We then convert this term frequency to the conditional distribution via Boltzmann distribution:

where $\gamma$ is temperature. We stochastically map the tokens to images with this conditional distribution $p(x_{i}\mid\mathit{tok})$. The results trained with these special vokens are shown in Table 5 as “Term Frequency”. Overall, token-level supervision is still better than the sentence-level supervision (as in the row “SentLabel”). However, among the models trained with token-level supervision, this token-level retrieval method neglects the contextual information thus is worse compared with sentence-level (in the row “Propagated”) and contextual token-level retrieval methods (in the row “Voken”) .

3 Visualization of Vokens

In Fig. 4, we visualize our generated vokens. The first example takes the leading sentence in our paper (without commas), which is also used in the imaginary example in Fig. 1. We also vokenize another sentence from William Yeats’s poet “Down by the Salley Gardens” in Fig. 4. Although the vokenizer is trained on image-captioning datasets without localizing token-to-image annotations, the vokenizer shows a strong selectivity: different images are selected w.r.t the tokens. The contextual token-level retrieval could also disambiguate certain tokens (e.g., “down” in Example 2) with the help of its context. When the unique related image is hard to define, our vokenizer aims to ground the non-concrete tokens (e.g., “by”/“and”/“the”) to relevant images: the voken for the token “by” in Example 2 (of Fig. 4) is better aligned with the [centering token, context] pair than the voken for the same token “by” in Example 1. This related visual information helps understand the language and leads to the improvement in Table 2. On the other hand, some tokens are not faithfully grounded (e.g., “writing” in Example 1) and we also observe a shift in alignment (e.g., the relevant image for the phrase “my love” in Example 2 is aligned to “my” instead of “love”). These misalignments are possibly caused by the limitations of sentence-image weak supervision in our training data since the strong token-image annotations are not available.

Related Work

Language pre-training has moved from token-level pre-training Mikolov et al. (2013); Pennington et al. (2014) to sentence-level pre-training Le and Mikolov (2014); Kiros et al. (2015); Conneau et al. (2017); Dai and Le (2015). Recently, a set of works Peters et al. (2018); Radford et al. (2018); Devlin et al. (2019); Yang et al. (2019); Liu et al. (2019); Clark et al. (2019); Lan et al. (2019) bring back token-level supervision with contextual language encoders (e.g., based on an LSTM Hochreiter and Schmidhuber (1997) and Transformers Vaswani et al. (2017)). This tendency inspires the design of our vokenizer in merging previous sentence-level Frome et al. (2013) and token-level Kiela et al. (2018) approaches into a contextual token-level approach.

Vision-and-Language Pre-training

Since language models are trained with self-supervision without knowing the connection to the visual world, vision-and-language pre-training Li et al. (2019); Lu et al. (2019); Tan and Bansal (2019); Chen et al. (2019); Su et al. (2020); Zhou et al. (2020) aims to build joint cross-modal representations and focuses on vision-and-language tasks. Due to particularity of grounded language, these models are not able to improve pure language tasks as shown in Sec. 5.1.

Visually-Aided Language Learning

Previous works use visual information to improve specific language tasks such as coreference resolution Kong et al. (2014), machine translation Elliott et al. (2016); Ive et al. (2019); Wu et al. (2019); Zhang et al. (2020), semantic parsing Christie et al. (2016); Shi et al. (2019); Kojima et al. (2020), and bilingual lexicon learning Kiela et al. (2015); Vulić et al. (2016). Our work has a focus on building a visually-supervised language pre-training frameworks to improve general language understanding. Similar to our work, Frome et al. (2013); Lazaridou et al. (2015); Collell et al. (2017); Kiela et al. (2018); Bordes et al. (2019) aim to improve language representation with visual information; however, most of these works focus on grounded language and hence might suffer from the large discrepancy that we discuss in Sec. 2.3.

Conclusion

In this paper, we explored the possibility of utilizing visual supervision to language encoders. In order to overcome the challenges in grounded language, we develop the vokenizer with contextual token-image matching models and use it to vokenize the language corpus. Supervised by these generated vokens, we observe a significant improvement over the purely self-supervised language model on multiple language tasks.

Acknowledgement

We thank the reviewers and Yixin Nie and Jie Lei for their helpful discussions. This work was supported by ARO-YIP Award W911NF-18-1-0336, DARPA MCS Grant N66001-19-2-4031, a Google Focused Research Award, and a Bloomberg Data Science Ph.D. Fellowship. The views, opinions, and/or findings contained in this article are those of the authors and not of the funding agency.

References

Appendix A Appendices

During the vokenization process, we use the faiss Johnson et al. (2019) library to speed up the nearest neighbor search. The vokenization runs at a speed of 100K tokens / second with 4 Titan V100 GPU. Thus the vokenization of the full Wikipedia is finished in 8 hours. When transferring vokens to other pre-training frameworks, revokenization does not need the GPU computation and runs as fast as the tokenization. The vokens are retrieved from the Visual Genome images which are not used in MS COCO (our training dataset). We take a voken size of $50000$.

When pre-training the model on pure language corpus, we unify the training process to avoid possible side effects from different training protocols. We follow previous work to conduct two simplifications: 1. Removing the next-sentence-prediction task Liu et al. (2019) 2. Using fixed sequence length Conneau et al. (2020) of $128$. We take the $12$-layer $\text{BERT}_{\text{BASE}}$ model of $768$ hidden dimensions and train it on English WikipediaDownloaded with https://github.com/attardi/wikiextractor for $200$K steps from scratch. We also take a reduced $6$-layer model and train it on Wiki103https://www.salesforce.com/products/einstein/ai-research/the-wikitext-dependency-language-modeling-dataset/ for $40$ epochs ($160$K steps) from scratch because this reduced model does not fit well on the full Wikipedia dataset. The voken classification task will not bring additional parameters to the language encoder (with $110$M parameters) but need more computations, we thus adjust the training steps for pure masked-language-model (MLM) training for a fair comparison. It results in around $10$% more training steps in pure MLM training. All models take batch sizes of $256$ and a learning rate of $2e\mbox{-}4$.

For fine-tuning tasks, instead of high-cost hyper-parameter sweeping in BERT Devlin et al. (2019), we train $3$ epochs with a learning rate of $1e\mbox{-}4$ and a batch-size of $32$ for all tasks in GLUE. The hyper-parameters for SQuAD and SWAG are borrowed from the BERT paper Devlin et al. (2019). On SQuAD v1.1, we fine-tune for $3$ epochs with a learning rate of $5e\mbox{-}5$ and a batch size of $32$. On SQuAD v2.0, we fine-tune for $2$ epochs with a learning rate of $5e\mbox{-}5$ and a batch size of $48$. On SWAG, we fine-tune for $3$ epochs with a learning rate of $2e\mbox{-}5$ and a batch size of $16$.

The whole framework is built on PyTorch Paszke et al. (2019). The implementations of BERT Devlin et al. (2019) and RoBERTa Liu et al. (2019) are borrowed from PyTorch Transformers Wolf et al. (2019) https://github.com/huggingface/transformers. All evaluation code is from the PyTorch Transformers as well.

A.2 Visually Grounded Token Types

In Sec.2.3, we estimate the visually grounded token types with the help of MS COCO Lin et al. (2014) dataset. We here randomly sample a list of the $2406$ grounded tokens used in the estimation:

photograph, tv, skyscraper, ##bery, wooded, little, stands, away, storage, mound, pouring, rail, ##fl, eye, ##ke, flown, skiing, plate, movie, dead, tossing, couple, racing, dust, licking, palm, stroll, granite, bananas, ledge, chained, monument, individuals, part, exhibit, softball, second, bow, ones, shop, beverages, sandy, sink, angle, ##ia, gives, music, leading, carrying, cookies, reading, faced, ##k, kid, ##ged, playing, winds, saddle, stunts, squat, cabinets, rusty, matching, biker, let, standing, pan, smiles, train, sky, passing, woman, military, feeder, lot, hydra, party, ##l, furnished, rides, strip, ##field, tin, crouched, courtyard, nicely, screens, us, lie, waving, process, equipment, structure, fore, barrier, ##li, beside, toast, catching, tracks

A.3 Maximum Inner Product Search of Norm-1 Vectors

In Sec. 3.1, we normalize the vector to norm-1 vectors thus the Maximum Inner Product Search Mussmann and Ermon (2016) is equivalent to Nearest Neighbor Knuth (1973). Here, we give a simple proof. Suppose $\bm{x}$ and $\bm{y}$ are two vectors of the same dimension, we have

A.4 Details of Sentence-level Retrieval in Analysis

In Sec. 3.1, we consider a contextual token-image matching model with relevance score $r_{\theta}(w,x;s)$. To do sentence-level retrieval, we modify it into a sentence-image matching score $r^{\prime}_{\theta}(x,s)$, and trained it with:

The score is also factorized as the dot product of the visual representation and the language representation. However, the language representation here is the sentence embedding (the output for the first token CLS).

These retrieved images as used as the label for the whole sentence.

A.5 Details of Token-level Retrieval in Analysis

In the purely token-level retrieval, we consider the image-captioning sentences as documents and uses traditional IR methods to index them. In order to increase the size of ‘documents’, we aggregate the data from VQA Antol et al. (2015) and Visual Genome Krishna et al. (2017), besides the existing MS COCO Lin et al. (2014) dataset. We also find that the temperature $\gamma\mbox{=}0.01$ gives a reasonable retrieval distribution and use it in our experiment.

A.6 Voken Ablation Studies

In Table 6, we show several approaches that provide alternative voken-like labels to our model.

Shuffle

In order to prove that the order of vokens would affect the results, we shuffle the vokens in each batch and use it as supervision.

Tokens

We here directly use the original tokens in replace of the vokens to see whether any dense supervision could improve the model.

As shown in Table 6, all these results are lower than the reference vokenization strategy.

A.7 Correlations between Improvements and Grounding Ratio

In order to understand where the improvements in the performance are coming from, we also study the correlation between the improvement in results and the visual grounding ratio (approximately measured in the same way as Sec. 2.3). We found that the datasets with higher grounding ratio (e.g., MNLI Williams et al. (2018)) get significant improvements while the datasets (e.g., QNLI Rajpurkar et al. (2016)) with relatively lower grounding ratio do not benefit much from the visual supervision. The dataset MNLI is built from multiple genre (the original SNLI dataset is in fact built from the Flickr images thus has a strong visual connection) and QNLI is purely based on English Wikipedia (The same as SQuAD Rajpurkar et al. (2016)). These correlations may indicate that the visual supervision helps build a better understanding of visually grounded tokens. Although we used contextual information to map non-grounded words to related images through vokenization, the effectiveness of this mapping relies on the original grounding ratio of the data.