Multilingual Autoregressive Entity Linking

Introduction

Entity Linking (EL, Hoffart et al., 2011; Dredze et al., 2010; Bunescu and Paşca, 2006; Cucerzan, 2007) is an important task in NLP, with plenty of applications in multiple domains, spanning Question Answering (De Cao et al., 2019; Nie et al., 2019; Asai et al., 2020), Dialogue (Bordes et al., 2016; Wen et al., 2017; Williams et al., 2017; Chen et al., 2017; Curry et al., 2018), Biomedical systems (Leaman and Gonzalez, 2008; Zheng et al., 2015), to name just a few. It consists of grounding entity mentions in unstructured texts to KB descriptors (e.g., Wikipedia articles).

The multilingual version of the EL problem has been for a long time tight to a purely cross-lingual formulation (XEL, McNamee et al., 2011a; Ji et al., 2015), where mentions expressed in one language are linked to a KB expressed in another (typically English). Recently, Botha et al. (2020) made a step towards a more inherently multilingual formulation by defining a language-agnostic KB, obtained by grouping language-specific descriptors per entity. Such formulation has the power of considering entities that do not have an English descriptor (e.g., a Wikipedia article in English) but have one in some other languages.

A common design choice to most current solutions, regardless of the specific formulation, is to provide a unified entity representation, either by collating multilingual descriptors in a single vector or by defining a canonical language. For the common bi-encoder approach (Wu et al., 2020; Botha et al., 2020), this might be optimal. However, in the recently proposed GENRE model (De Cao et al., 2021), an autoregressive formulation to the EL problem leading to stronger performance and considerably smaller memory footprints than bi-encoder approaches on monolingual benchmarks, the representations to match against are entity names (i.e., strings) and it’s unclear how to extend those beyond a monolingual setting.

In this context, we find that maintaining as much language information as possible, hence providing multiple representations per entity, helps due to the connections between source language and entity names in different languages. We additionally find that using all available languages as targets and aggregating over the possible choices is an effective way to deal with a zero-shot setting where no training data is available for the source language.

Concretely, in this paper, we present mGENRE, the first multilingual EL system that exploits a sequence-to-sequence architecture to generate entity names in more than 100 languages left to right, token-by-token in an autoregressive fashion and conditioned on the context (see Figure 1 for an outline of our system). While prior works use a single representation for each entity, we maintain entity names for as many languages as possible, which allows exploiting language connections between source input and target name. To summarize, this work makes the following contributions:

Extend the catalog of entity names by considering all languages for each entry in the KB. Storing the multilingual names index is feasible and cheap (i.e., 2.2GB for $\sim$89M names).

Design a novel objective function that marginalizes over all languages to perform a prediction. This approach is particularly effective in dealing with languages not seen during fine-tuning ($\sim$50% improvements).

Establish new state-of-the-art performance for the Mewsli-9 (Botha et al., 2020), TR2016hard (Tsai and Roth, 2016a) and TAC-KBP2015 (Ji et al., 2015) MEL datasets.

Present extensive analysis of modeling choices, including the usage of candidates from a mention table, frequency-bucketed evaluation, and performance on an heldout set including low-resource languages.

Publicly release our best model, pre-trained as multilingual denoising auto-encoder using the BART objective Lewis et al. (2020); Liu et al. (2020) on large-scale monolingual corpora in 125 languages and fine-tuned to generate entity names given $\sim$730M in-context Wikipedia hyperlinks in 105 languages.

Background

We first introduce Multilingual Entity Linking in Section 2.1 highlighting its difference with monolingual and cross-lingual linking. We address the MEL problem with a sequence-to-sequence model that generates textual entity identifiers (i.e., entity names). Our formulation generalizes the GENRE model by De Cao et al. (2021) to a multilingual setting (mGENRE). Thus in Section 2.2 and 2.3, we discuss the GENRE model and how it ranks entities with Beam Search respectively.

Multilingual Entity Linking (MEL, Botha et al., 2020) is the task of linking a given entity mention $m$ in a given context $c$ of language $l\in\mathcal{L}_{C}$ to the corresponding entity $e\in\mathcal{E}$ in a multilingual Knowledge Base (KB). See Figure 1 for an example: there are textual inputs with entity mentions (in bold) and we ask the model to predict the corresponding entities in the KB. A language-agnostic KB includes at least the name (but could include also descriptions, aliases, etc.) of each entity in one or more languages but there is no assumption about the relationship between these languages $\mathcal{L}_{KB}$ and languages of the context $\mathcal{L}_{C}$. This is a generalization of both monolingual Entity Linking EL and cross-lingual EL (XEL, McNamee et al., 2011a; Ji et al., 2015). The latter considers contexts in different languages while mapping mentions to entities in a monolingual KB (e.g., English Wikipedia).

Additionally, we assume that each $e\in\mathcal{E}$ has a unique textual identifier in at least a language. Concretely, in this work, we use Wikidata (Vrandečić and Krötzsch, 2014) as our KB. Each Wikidata item lists a set of Wikipedia pages in multiple languages linked to it and in any given language each page has a unique name (i.e., its title).

2 Autoregressive generation

3 Ranking with Constrained Beam Search

At test time, it is prohibitively expensive to compute a score for every element in $\mathcal{E}$ and then sort them. Thus, GENRE exploits Beam Search (BS, Sutskever et al., 2014), an established approximate decoding strategy to navigate the search space efficiently. Instead of explicitly scoring all entities in $\mathcal{E}$, search for the top-$k$ entities in $\mathcal{E}$ using BS with $k$ beams. BS only considers one step ahead during decoding (i.e., it generates the next token conditioned on the previous ones). Thus, GENRE employs a prefix tree (trie) to enable constrained beam search and then generate only valid entity identifiers.

Model

To extend GENRE to a multilingual setting we need to define what are the unique identifiers of all entities in a language-agnostic fashion. This is not trivial since we rely on text representations that are by their nature grounded in some language. Concretely, for each entity $e$, we have a set of identifiers $\mathcal{I}_{e}$ that consists of pairs $\langle l,n_{e}^{l}\rangle$ where $l\in\mathcal{L}_{KB}$ indicates a language and $n_{e}^{l}$ the name of the entity $e$ in the language $l$. We extracted these identifiers from our KB—each Wikidata item has a set of Wikipedia pages in multiple languages linked to it, and in any given language, each page has a unique name. We identified 3 strategies to employ these identifiers:

define a canonical textual identifier for each entity such that there is a 1-to-1 mapping between the two (i.e., for each entity, select a specific language for its name—see Section 3.1);

define a N-to-1 mapping between textual identifier and entities concatenating a language ID (e.g., a special token or the ISO 639-1 codehttps://www.iso.org/standard/22109.html) followed by its name in that language—alternatively concatenating its name first and then a language ID (see Section 3.2);

treat the selection of an identifier in a particular language as a latent variable (i.e., we let the model learn a conditional distribution of languages given the input and we marginalize over those—see Section 3.3).

All of these strategies define a different way we compute the underlining likelihood of our model. In Figure 1 we show an outline of mGENRE. The following subsections will present detailed discussions of the above 3 strategies.

2 Multilingual entity representation

The former corresponds to first predicting a distribution over languages and then predicting a title conditioning on the language $l$ where the latter corresponds to the opposite. Predicting the language first conditions the generation to a smaller set earlier during beam search (i.e., all names in a specific language). However, it might exclude some targets from the search too early if the beam size is too small. Predicting the language last does not condition the generation of names in a particular language but it asks the model to disambiguate the language of the generated name whenever it is ambiguous (i.e., when the same name in different languages corresponds to possibly different entities). Only 1.65% of the entity names need to be disambiguated with the language. In practice, we observe no difference in performance between the two approaches. Both strategies define an N-to-1 mapping between textual identifiers and entities and then at test time we just use a lookup table to select the correct KB item. This N-to-1 mapping is an advantage compared to using canonical names because the model can predict in any available language and therefore exploit synergies between source and target language as well as avoiding translation.

3 Marginalization

4 Candidate selection

Modern EL systems that employ cross-encoding between context and entities usually do not score all entities in a KB as it is too computational expensive (Wu et al., 2020). Instead, they first apply candidate selection (with a less expensive method first or just a non-parametric mention table) to reduce the number of entities before scoring. In our formulation, there is no need to do that since mGENRE uses Beam Search to efficiently generate. However, using candidates might help, and therefore, we also experiment with that. Scoring all candidates might not be always possible (sometimes there are thousands of candidates for a mention) and especially when using an N-to-1 mapping between textual identifiers there will be names to rank in all languages available for each candidate. Thus, when we use candidates, it is to constrain BS steps further, rather than to rank all of them.

Concretely, candidate selection is made with an alias table. Using the training data, we build a mention table where we record all entities indexed by the names used to refer to them. Additionally, we also use Wikipedia titles as additional mentions (useful for entities that never appear as links), redirects, Wikidata labels, and aliases.

Experimental Setting

We use Wikidata (Vrandečić and Krötzsch, 2014) as our KB while exploiting the supervision signal from Wikipedia hyperlinks. For evaluation, we test our model on two established cross-lingual datasets, TR2016hard and TAC-KBP2015 (Ji et al., 2015; Tsai and Roth, 2016a), as well as the recently proposed Mewsli-9 MEL dataset (Botha et al., 2020). Additionally, we propose a novel setting extracted from Wikinewshttps://www.wikinews.org where we train a model on a set of languages, and we test it on unseen ones.

We use Wikidata as the target KB to link to filtering the same heuristic as Botha et al. (2020) (see Appedinx A for more details). Eventually, our entity set $\mathcal{E}$ contains 20,277,987 items (as a reference, English Wikipedia has just $\sim$6M items). Using the corresponding Wikipedia titles as textual identifiers in all languages leads to a table of 53,849,351 entity names. We extended the identifiers including redirects which leads to a total of 89,270,463 entity names (see Table 10 in Appendix A for more details). Although large, the number of entity names is not a bottleneck as the generated prefix tree only occupies 2.2GB for storage. As a comparison the Botha et al.’s (2020) MEL systems need $\sim$10 times more storage.

2 Supervision: Wikipedia

For all experiments, we do not train a model from scratch, but we fine-tune a multilingual language model trained on 125 languages (see Appendix A for more details on the pre-trained model). We exploit Wikipedia hyperlinks as the source of supervision for MEL. We used Wikipedia in 105 languages out of the $>$300 available. These 105 are all the languages for which our model was pre-trained on that overlaps with the one available in Wikipedia (see full language list in Figure 2 and more details in Appendix A). Eventually, we extracted a large-scale dataset of 734,826,537 datapoints. For the plain generation strategy, we selected as the ground truth the name in the source language. When such entity name is not available we randomly select 5 alternative languages and we use all of them as datapoints. To enable model selection, we randomly selected 1k examples from each language to keep as a validation set.

3 Datasets

(Botha et al., 2020) contains 289,087 entity mentions appearing in 58,717 originally written news articles from Wikinews, linked to WikiData. The corpus includes documents in 9 languages.Arabic, English, Farsi, German, Japanese, Serbian, Spanish, Tamil, and Turkish. Differently from the cross-lingual setting, this is a truly multilingual dataset since 11% target entities in Mewsli-9 do not have an English Wikipedia page thus a XEL model would not link these.

TR2016hard

(Tsai and Roth, 2016a) is a Wikipedia based cross-lingual dataset specifically constructed to contain difficult mention-entity pairs. Authors extracted Wikipedia hyperlinks for which the corresponding entity is not the most likely when using an alias table. Since we train on Wikipedia, to avoid an overlap with this test data, we removed all mentions from our training data that also appear in TR2016hard. Note that this pruning strategy is more aggressive than Tsai and Roth’s (2016a) and Botha et al.’s (2020) strategies. Tsai and Roth (2016a) assured to not have mention-entity pairs overlaps between training and test, but a mention (with a different entity) might appear in training. Botha et al. (2020)Information provided by private correspondence with the authors. split at the page-level only, making sure to hold out all Tsai and Roth (2016a) test pages (and their corresponding pages in other languages), but they trained on any mention-entity pair that could be extracted from their remaining training page partition (i.e., they have overlap between training and text entity-mention pairs). To compare with previous works (Tsai and Roth, 2016a; Upadhyay et al., 2018; Botha et al., 2020) we only evaluate on German, Spanish, French and Italian (a total of 16,357 datapoints).

TAC-KBP2015

To evaluate our system on documents out of the Wikipedia domain, we experiment on the TAC-KBP2015 Tri-Lingual Entity Linking Track (Ji et al., 2015). To compare with previous works (Tsai and Roth, 2016a; Upadhyay et al., 2018; Sil et al., 2018; Zhou et al., 2019), we use only Spanish and Chinese (i.e., we do not evaluate in English). Following previous work, we only evaluate in-KB links (Yamada et al., 2016; Ganea and Hofmann, 2017), i.e, we do not evaluate on mentions that link to entities out of the KB. Previous works considered Freebase (Bollacker et al., 2008) as KB, and thus we computed a mapping between Freebase ID and Wikidata ID. When we cannot solve the match, our system gets zero scores (i.e., it counts as a wrong prediction). TAC-KBP2015 contains 166 Chinese documents (84 news and 82 discussion forum articles) and 167 Spanish documents (84 news and 83 discussion forum articles) for a total of 12,853 mention-entity datapoints.

Wikinews-7

For the purpose of testing a model on languages unseen during training, we extract mention-entities pairs from Wikinews in 7 languages that are not in the Mewsli-9 language set.Chinese, Czech, French, Italian, Polish, Portuguese, and Russian. Table 7 in Appendix A reports statistics of this dataset. Wikinews-7 is created in the same way as Mewsli-9, but we used our own implementation to extract data from raw dumpsBotha et al. (2020) did not release code for extracting Mewsli-9 from a Wikinews dump..

Results

The main results of this work are reported in Table 1 for Mewsli-9, and in Table 2 for TR2016hard, and TAC-KBP2015 respectively. Our mGENRE (trained with ‘title+lang’) outperforms all previous works in all those datasets. We show the accuracy of mGENRE on the 105 languages in our Wikipedia validation set against an alias table baseline in Figure 2. In Table 11 in Appendix A we report more details on the validation set results. In Table 5 we report some examples of correct and wrong predictions of our mGENRE on selected datapoints from Mewsli-9.

In Table 1 we compare our mGENRE against the best model from Botha et al. (2020) (Model F+) as well as with their alias table baseline. We report results from mGENRE with and without constraining the beam search to the top-k candidates from the table (see Section 3.4) as well as with and without marginalization (see Section 3.3). All of these alternatives outperform Model F+ on both micro and macro average accuracy across the 9 languages. Our base model (without candidates or marginalization) has a 10.9% error reduction in micro average and 18.0% error reduction for macro average over all languages. The base model has no restrictions on candidates so it is effectively classifying among all the $\sim$20M entities. The base model performs better than Model F+ on each individual language except English and German. Note that these languages are the ones for which we have more training data ($\sim$134M and $\sim$60M datapoints each) but also the languages that have the most entities/pages ($\sim$6.1M and $\sim$2.4M). Therefore these are the hardest languages to link.Even though there are many datapoints, disambiguating between more entities is harder. Moreover, low-resource languages should be harder as there are fewer training instances, but there are no such languages in the Mewsli-9 set. When enabling candidate filtering to restrict the space for generation, we further improve error reduction to 13.6% and 21.0% for micro and macro average respectively. Marginalization reduces the error by the same amount as candidate filtering but combining search with candidates and marginalization leads to our best model: it improves error reduction to 14.5% and 23.0% on micro and macro average respectively. Our best model is also better than Model F+ in English and on par with it in German.

TR2016hard and TAC-KBP2015

We compared our mGENRE against cross-lingual systems (Tsai and Roth, 2016a; Sil et al., 2018; Upadhyay et al., 2018; Zhou et al., 2019) and Model F+ by Botha et al. (2020) in Table 2. Differently from Meswli-9, the base mGENRE model does not outperform previous systems. Using marginalization brings minimal improvements. Instead, using candidates gives +11% absolute accuracy on TAC-KBP2015 and +5% on TR2016hard effectively making mGENRE state-of-the-art in both datasets. The role of candidates is very evident on TAC-KBP2015 where there is not much of a difference for Spanish but a +22% absolute accuracy for Chinese. TAC-KBP2015 comes with a training set and we used it to expand the candidate set. Additionally, we also included all simplified Chinese versions of the entity names because we used traditional Chinese in pre-training, and TAC-KBP2015 uses simplified Chinese. Many mentions in TAC-KBP2015 were not observed in Wikipedia, so the performance gain mostly comes from this but including the simplified and alternative Chinese names also played an important role (+5% comes from this alone).We speculate that including different version (e.g., different dialects for Arabic) of entity names could improve performance in all languages. Since this is not in the scope of this paper, we will leave it for future work.

2 Analysis

Table 3 shows a breakdown of Mewsli-9 accuracy by entity frequency in training for Botha et al.’s (2020) Model F+ and mGENRE. Interestingly, our mGENRE has much higher accuracy (22% vs 8%) on unseen entities (i.e., the [0,1) bin). This is because our formulation can take advantage of copying names from the source, translating them or normalizing them. For example, an unseen person name should likely be linked to the entity with the same name. This is a powerful bias that gives the model advantage in these cases. On very rare entities (i.e., the [1,10) bin) our model performs worse than Model F+. Note that Model F+ was trained specifically to tackle those cases (e.g., with hard negatives and frequency-based mini-batches) whereas our model was not. We argue that similar strategies can be applied to mGENRE to improve performance on rare entities, and we leave that to future work. The performance gap between Model F+ and mGENRE on entities that appear more than 100 times in the training set is negligible.

By candidate frequency

We additionally measure the accuracy on Mewsli-9 by the number of candidates retrieved from the alias table (details in Figure 3). When there are no candidates ($\sim$12k datapoints that is $\sim$4% Mewsli-9) an alias table would automatically fail, but mGENRE uses the entire KB as candidates and has 63.9% accuracy. For datapoints with few candidates (e.g., less than 100), we could use mGENRE as a ranker and score all of the options without relying on constrained beam search. However, this approach would be computationally infeasible when there are no candidates (i.e., we use all the KB as candidates) or too many candidates (e.g., thousands). Constrained BS allows us to efficiently explore the space of entity names, whatever the number of candidates.

Unseen Languages

We use our Wikinews-7 dataset to evaluate mGENRE capabilities to deal with languages not seen during training (i.e., the set of languages in train and test are disjoint). This zero-shot setting implies that no mention table is available during inference; hence we do not consider candidates for test mentions. We train our models on the nine Mewsli-9 languages and compare all strategies exposed in Section 3. To make our ablation study feasible, we restrict the training data to the first 1 million hyperlinks from Wikipedia abstracts. Results are reported in Table 4.

Using our novel marginalization strategy that aggregates (both at training and inference time) over all seen languages to perform the linking brings an improvement of over 50% with respect to considering a single language. To deeper investigate the behaviour of the model in this setting, we compute the probability mass distribution over languages seen at training time for the first prediction (reported in Figure 4). When marginalization is enabled (Figure 4(b)) the distribution is more spread across languages since the model is trained to use all of them. Hence the model can exploit connections between an unseen language and all seen languages for the linking process, which drastically increases the accuracy.

Related Work

The most related works to ours are De Cao et al. (2021), that proposed to use an autoregressive language model for monolingual EL, and Botha et al. (2020) that proposes to extend the cross-lingual EL task to multilingual EL with a language-agnostic KB. We provide an outline of the GENRE model proposed by De Cao et al. (2021) in Section 2.2 and 2.3. GENRE was applied not only to EL but also for joint mention detection and entity linking (still with an autoregressive formulation) as well as to page-level document retrieval across multiple Knowledge Intensive Language Tasks (KILT; Petroni et al., 2021) i.e., fact-checking, open-domain question answering, slot filling, and dialog. Botha et al.’s (2020) Model F+ is a bi-encoder model: it is based on two BERT-based (Devlin et al., 2019) encoders that outputs vector representations for contet and entities. Similar to Wu et al. (2020) they rank entities with a dot-product between these representations. Model F+ uses the description of entities as input to the entity encoder and title, document and mention (separated with special tokens) as inputs to the context encoder. Bi-encoders solutions may be memory inefficient since they require to keep in memory big matrices of embeddings, although memory-efficient dense retrieval has recently received attention (Izacard et al., 2020; Min et al., 2021; Lewis et al., 2021).

Another widely explored line of work is Cross-Language Entity Linking (XEL; McNamee et al., 2011b; Cheng and Roth, 2013). XEL considers contexts in different languages while mapping mentions to entities in a monolingual KB (e.g., English Wikipedia). Tsai and Roth (2016b) used alignments between languages to train multilingual entity embeddings. They used candidate selection and then they re-rank them with an SVM using these embeddings as well as a set of features (based on the multilingual title, mention, and context tokens). Sil et al. (2018) explored the use of more sophisticated neural models for XEL as well as Upadhyay et al. (2018) who jointly modeled type information to boost performance. Zhou et al. (2019) propose improvements to both entity candidate generation and disambiguation to make better use of the limited data in low-resource scenarios. Note that in this work we focus on multilingual EL, not cross-lingual. XEL is limiting to a monolingual KB (usually English), where MEL is more general since it can link to entities that might not be necessary represented in the target monolingual KB but in any of the available languages.

Conclusion

In this work, we propose an autoregressive formulation to the multilingual entity linking problem. For a mention in a given language, our solution generates entity names left-to-right and token-by-token. The resulting system, mGENRE, maintains entity names in as many languages as possible to exploit language connections and interactions between source mention context and target entity name. The constrained beam search decoding strategy enables fast search within a large set of entity names (e.g., the whole KB in multiple languages) with no need for large-scale dense indices. We additionally design a novel objective function that marginalizes over all available languages to perform a prediction. We empirically show that this strategy is really effective in dealing with languages for which no training data is available (i.e., 50% improvements for languages never seen during training). Overall, our experiments show that mGENRE achieves new state-of-the-art performance on three popular multilingual entity linking datasets.

Acknowledgments

Authors thank Patrick Lewis, Aleksandra Piktus, for helpful discussions and technical support.

References

Appendix A Experimental Details

We used a pre-trained mBART (Lewis et al., 2020; Liu et al., 2020) model on 125 languages—see Figure 5 for a visual overview of the overlap with these languages, Wikipedia and the languages used by Botha et al. (2020). mBART has 24 layers of hidden size is 1,024 and it has a total of 406M parameters. We pre-trained on an extended version of the cc100 (Conneau et al., 2020; Wenzek et al., 2020) corpora available herehttp://data.statmt.org/cc-100 where we increased the number of common crawl snapshots for low resource languages from 12 to 60. The dataset has $\sim$5TB of text. We pre-trained for 500k steps with max 1,024 tokens per GPU on a variable batch size ($\sim$3000). Figure 5 shows a Venn diagram on the overlap of languages used during pre-training and fine-tuning.

A.2 Data for supervision

Wikidata contains tens of millions of items but most of them are scholarly articles or they correspond to help and template pages in Wikipedia (i.e., not entities we want to retain) https://www.wikidata.org/wiki/Wikidata:Statistics. Following (Botha et al., 2020), we only keep Wikidata items that have an associated Wikipedia page in at least one language, independent of the languages we actually model. Moreover, we filter out items that are a subclass (P279) or instance of (P31) some Wikimedia organizational entities (e.g., help and template pages—see Table 6).

Wikipedia

We aligned each Wikipedia hyperlink to its respective Wikidata item using a custom script. Note that each Wikipedia page maps to a Wikidata item. For the alignment we use i) direct reference when the hyperlink point directly to a Wikipedia page, ii) a re-directions table if the hyperlink points to an alias page, and iii) a Wikidata search among labels and aliases of items if the previous two alignment strategies failed. The previous two alignment strategies might fail when i) authors made a mistake linking on a non-existing page, ii) authors linked to a non-existing page on purpose hoping it will be created in the future, or iii) the original title of a page changed over time and no redirection was added to accommodate old hyperlinks. This procedure successfully aligns 91% of the hyperlinks. We only keep unambiguous alignments since, when using Wikidata search (i.e., the third alignment strategy), the mapping could be ambiguous (e.g., multiple items may share the same labels and aliases).

In Table 10 we report some statistics of the training data extracted from Wikipedia. We use a standard Wikipedia extractor wikiextractorhttps://github.com/attardi/wikiextractor by Attardi (2015) and a redirect extractorhttps://code.google.com/archive/p/wikipedia-redirect. We use both Wikipedia and Wikidata dumps from 2019-10-01.

A.3 Data for test

We use Wikinews (from 2019-10-01) to construct our unseen Wikinews-7 dataset. In Table 7 we report some statistic of our new dataset.

A.4 Training

We implemented, trained, and evaluate our model using the fariseq library (Ott et al., 2019). We trained mGENRE using Adam (Kingma and Ba, 2014) with a learning rate $10^{-4}$, $\beta_{1}=0.9$, $\beta_{2}=0.98$, and with a linear warm-up for 5,000 steps followed by liner decay for maximum 2M steps. The objective is sequence-to-sequence categorical cross-entropy loss with 0.1 of label smoothing and 0.01 of weight decay. We used dropout probability of 0.1 and attention dropout of 0.1. We used max 3,072 tokens per GPU and variable batch size ($\sim$12,500). Training was done on 384 GPUs (Tesla V100 with 32GB of memory) and it completed in $\sim$72h for a total of $\sim$27,648 GPU hours or $\sim$1,152 GPU days. Since TAC-KBP2015 contains noisy text (e.g., XML/HTML tags), we further fine-tune mGENRE for 2k steps on its training set when testing on it.

A.5 Inference

At test time, we use Constrained Beam Search with 10 beams, length penalty of 1, and maximum decoding steps of 32. We restrict the input sequence to be at most 128 tokens cutting the left, right, or both parts of the context around a mention. When employing marginalization, we normalize the log-probabilities by sequence length using $\log p(y|x)/L^{\alpha}$, where $\alpha=0.5$ was tuned on the development set.

Appendix B Additional results

We show a breakdown of the accuracy of mGENRE on Mewsli-9 by mention frequency in Table 8. The accuracy of unseen mentions is 66.7% and increases up to 93.6% for mentions seen more than 10k times. For extremely common mentions (i.e., seen more than 1M times) the accuracy drops to 73.2%. These mentions correspond to entities that are harder to disambiguate (e.g., ‘United States’ appears 3.2M times but can be linked to the country as well as any sports team where the context refers to sports).

Unseen Languages

Even though marginalization and canonical representation are the top-two systems in the unseen languages setting, they are not on seen languages. In Table 9 we report the results of all these strategies also on the seen languages (Mewsli-9 test set). Complementary to Figure 4 we also report the probability mass distribution over languages seen for Mewsli-9.