Towards Better UD Parsing: Deep Contextualized Word Embeddings, Ensemble, and Treebank Concatenation

Introduction

In this paper, we describe our system (HIT-SCIR) submitted to CoNLL 2018 shared task on Multilingual Parsing from Raw Text to Universal Dependencies Zeman et al. (2018). We base our system on Stanford’s winning system (Dozat et al., 2017, §2) for the CoNLL 2017 shared task Zeman et al. (2017).

Dozat and Manning (2016) and its extension Dozat et al. (2017) have shown very competitive performance in both the shared task Dozat et al. (2017) and previous parsing works Ma and Hovy (2017); Shi et al. (2017a); Liu et al. (2018b); Ma et al. (2018). A natural question that arises is how can we further improve their part of speech (POS) tagger and dependency parser via a simple yet effective technique. In our system, we make two noteworthy extensions to their tagger and parser:

Incorporating the deep contextualized word embeddings (Peters et al., 2018, ELMo: Embeddings from Language Models) into the word representaton (§3);

Ensembling parsers trained with different initialization (§4).

For some languages in the shared task, multiple treebanks of different domains are provided. Treebanks which are of the same language families are provided as well. Letting these treebanks help each other has been shown an effective way to improve parsing performance in both the cross-lingual-cross-domain parsing community and last year’s shared tasks Ammar et al. (2016); Guo et al. (2015); Che et al. (2017); Shi et al. (2017b); Björkelund et al. (2017). In our system, we apply the simple concatenation to the treebanks that are potentially helpful to each other and explore different ways of concatenation to improve the parser’s performance (§5).

In dealing with the small treebanks and treebanks from low-resource languages (§6), we adopt the word embedding transfer idea in the cross-lingual dependency parsing Guo et al. (2015) and use the bilingual word vectors transformation technique Smith et al. (2017)https://github.com/Babylonpartners/fastText_multilingual to map fasttexthttps://github.com/facebookresearch/fastText word embeddings Bojanowski et al. (2016) of the source rich-resource language and target low-resource language into the same space. The transferred parser trained on the source language is used for the target low-resource language.

We conduct experiments on the development data to study the effects of ELMo, parser ensemble, and treebank concatenation. Experimental results show that these techniques substantially improve the parsing performance. Using these techniques, our system achieved an averaged LAS of 75.84 on the official test set and was ranked the first according to LAS Zeman et al. (2018). This result significantly outperforms the others by a large margin.http://universaldependencies.org/conll18/results.html

We release our pre-trained ELMo for many languages at https://github.com/HIT-SCIR/ELMoForManyLangs.

Deep Biaffine Parser

We based our system on the tagger and parser of Dozat et al. (2017). The core idea of the tagger and parser is using an LSTM network to produce the vector representation for each word and then predict POS tags and dependency relations using the representation. For the tagger whose input is the word alone, this representation is calculated as

where $\mathbf{v}_{i}^{(word)}$ is the word embeddings. After getting $\mathbf{h}_{i}$, the scores of tags are calculated as

where each element in $\mathbf{s}_{i}^{(pos)}$ represents the possibility that $i$-th word is assigned with corresponding tag.

For the parser whose inputs are the word and POS tag, such representation is calculated as

And a pair of representations are fed into a biaffine classifier to predict the possibility that there is a dependency arc between these two words. The scores over all head words are calculated as

where $\mathbf{h}_{i}^{(arc\text{-}dep)}$ is computed by feeding $\mathbf{h}_{i}$ into an MLP and $H^{(arc\text{-}head)}$ is the stack of $\mathbf{h}_{i}^{(arc\text{-}head)}$ which is calculated in the same way as $\mathbf{h}_{i}^{(arc\text{-}dep)}$ but using another MLP. After getting the head $y^{(arc)}$ word, its relation with $i$-th word is decided by calculating

where $\mathbf{h}^{(rel-head)}$ and $\mathbf{h}^{(rel-dep)}$ are calculated in the same way as $\mathbf{h}_{i}^{(arc\text{-}dep)}$ and $\mathbf{h}_{i}^{(arc\text{-}head)}$.

This decoding process can lead to cycles in the result. Dozat et al. (2017) employed an iterative fixing methods on the cycles. We encourage the reader of this paper to refer to their paper for more details on training and decoding.

For both the biaffine tagger and parser, the word embedding $\mathbf{v}_{i}^{(word)}$ is obtained by summing a fine-tuned token embedding $\mathbf{w}_{i}$, a fixed word2vec embedding $\mathbf{p}_{i}$, and an LSTM-encoded character representation $\mathbf{\hat{v}}_{i}$ as

Deep Contextualized Word Embeddings

Deep contextualized word embeddings (Peters et al., 2018, ELMo) has shown to be very effective on a range of syntactic and semantic tasks and it’s straightforward to obtain ELMo by using an LSTM network to encode words in a sentence and training the LSTM network with language modeling objective on large-scale raw text. More specifically, the $\mathbf{ELMo}_{i}$ is computed by first computing the hidden representation $\mathbf{h}_{i}^{(LM)}$ as

In this paper, we study the usage of ELMo for improving both the tagger and parser and make several simplifications. Different from Peters et al. (2018), we treat the output of ELMo as a fixed representation and do not tune its parameters during tagger and parser training. Thus, we cancel the layer-wise attention scores $s_{j}$ and the scaling factor $\gamma$, which means

In our preliminary experiments, using $\mathbf{h}_{i,0}^{(LM)}$ for $\mathbf{ELMo}_{i}$ yields better performance on some treebanks. In our final submission, we decide using either $\sum_{j=0}^{2}\mathbf{h}_{i,j}^{(LM)}$ or $\mathbf{h}_{i,0}^{(LM)}$ based on their development.

After getting $\mathbf{ELMo}_{i}$, we project it to the same dimension as $\mathbf{v}_{i}^{(word)}$ and use it as an additional word embedding. The calculation of $\mathbf{v}_{i}^{(word)}$ becomes

for both the tagger and parser. We need to note that training the tagger and parser includes $W^{(ELMo)}$. To avoid overfitting, we impose a dropout function on projected vector $W^{(ELMo)}\cdot\mathbf{ELMo}_{i}$ during training.

Parser Ensemble

According to Reimers and Gurevych (2017), neural network training can be sensitive to initialization and Liu et al. (2018a) shows that ensemble neural network trained with different initialization leads to performance improvements. We follow their works and train three parsers with different initialization, then ensemble these parsers by averaging their softmaxed output scores as

Treebank Concatenation

For 15 out of the 58 languages in the shared task, multiple treebanks from different domains are provided. There are also treebanks that come from the same language family. Taking the advantages of the relation between treebanks has been shown a promising direction in both the research community Ammar et al. (2016); Guo et al. (2015, 2016a) and in the CoNLL 2017 shared task Che et al. (2017); Björkelund et al. (2017); Shi et al. (2017b). In our system, we adopt the treebank concatenation technique as Ammar et al. (2016) with one exception: only a group of treebanks from the same language (cross-domain concatenation) or a pair of treebanks that are typologically or geographically correlated (cross-lingual concatenation) is concatenated.

In our system, we tried cross-domain concatenation on nl, sv, ko, it, en, fr, gl, la, ru, and sl.We opt out cs, fi, and pl because all the treebanks of these languages are relatively large – they have more than 10K training sentences. We also tried cross-lingual concatenation on ug-tr, uk-ru, ga-en, and sme-fi following Che et al. (2017). However, due to the variance in vocabulary, grammatical genre, and even annotation, treebank concatenation does not guarantee to improve the model’s performance. We decide the usage of concatenation by examining their development set performance. For some small treebanks which do not have development set, whether using treebank concatenation is decided through 5-fold cross validation.We use udpipe for this part of experiments because we consider the effect of treebank concatenation as being irrelevant to the parser architecture and udpipe has the speed advantage in both training and testing. We show the experimental results of treebank concatenation in Section 9.3.

Low Resources Languages

In the shared task, 5 languages are presented with training set of less than 50 sentences. 4 languages do not even have any training data. It’s difficult to train reasonable parser on these low-resource languages. We deal with these treebanks by adopting the word embedding transfer idea of Guo et al. (2015). We transfer the word embeddings of the rich-resource language to the space of low-resource language using the bilingual word vectors transformation technique Smith et al. (2017) and trained a parser using the source treebank with only pretrained word embeddings on the transformed space as $\mathbf{v}_{i}^{(word)}=\mathbf{p}_{i}$. The transformation matrix is automatically learned on the fasttext word embeddings using the same tokens shared by two languages (like punctuation).

Table 1 shows our source languages for the target low-resource languages. For a treebank with a few training data, its source language is decided by testing the source parser’s performance on the training data.We use udpipe for this test. When training the parser, the small set of target training data is also used. For a treebank without any training data, we choose the source language according to their language family.Thai does not have a treebank in the same family. We choose Chinese as source language because of geographical closeness and both these two languages are SVO in typology.

Naija presents an exception for our method since it does not have fasttext word embeddings and embedding transformation is infeasible. Since it’s a dialect of English, we use the full pipeline of en_ewt for pcm_nsc instead.

Preprocessing

Besides improving the tagger and parser, we also consider the preprocessing as an important factor to the final performance and improve it by using the state-of-the-art system for sentence segmentation, or developing our own word segmentor for languages whose tokenizations are non-trival.

For some treebanks, sentence segmentation can be problematic since there is no explicitly sentence delimiters. de Lhoneux et al. (2017) and Shao (2017) presented a joint tokenization and sentence segmentation model (denoted as Uppsala segmentor)https://github.com/yanshao9798/segmenter/ that outperformed the baseline model in last year’s shared task Zeman et al. (2017). We select a set of treebanks whose udpipe sentence segmentation F-scores are lower than 95 on the development set and use Uppsala segmentor instead.We use Uppsala segmentor for it_postwita, got_proiel, la_poroiel, cu_proiel, grc_proiel, sl_ssj, nl_lassysmall, fi_tdt, pt_bosque, da_ddt, id_gsd, el_gdt, and et_edt. Using the Uppsala segmentor leads to a development improvement of 7.67 F-score in these treebanks over udpipe baseline and it was ranked the first according to sentence segmentation in the final evaluation.

2 Tokenization for Chinese, Japanese, and Vietnamese

Tokenization is non-trivial for languages which do not have explicit word boundary markers, like Chinese, Japanese, and Vietnamese. We develop our own tokenizer (denoted as SCIR tokenizer) for these three languages. Following Che et al. (2017) and Zheng et al. (2017), we model the tokenization as labeling the word boundary tagWe use the BIES scheme. on characters and use features derived from large-scale unlabeled data to further improve the performance.For Vietnamese where whitespaces occur both inter- and intra-words, we treat the whitespace-separated token as a character. In addition to the pointwise mutual information (PMI), we also incorporate the character ELMo into our tokenizer. Embeddings of these features are concatenated along with a bigram character embeddings as input. These techniques lead to the best tokenization performance on all the related treebanks and the average improvement over udpipe baseline is 7.5 in tokenization F-score.on ja_gsd, ja_modern, vi_vtb, and zh_gsd.

3 Preprocessing for Thai

Thai language presents a unique challenge in the preprocessing. Our survey on the Thai Wikipedia indicates that there is no explicit sentence delimiter and obtaining Thai words requires tokenization. To remedy this, we use the whitespace as sentence delimiter and use the lexicon-based word segmentation – forward maximum matching algorithm for Thai tokenization. Our lexicon is derived from the fasttext word embeddings by preserving the top 10% frequent words.

4 Lemmatization and Morphology Tagging

We did not make an effort on lemmatization and morphology tagging, but only use the baseline model. This lags our performance in the MLAS and BLEX evaluation, in which we were ranked 6th and 2nd correspondingly. However, since our method, especially incorporating ELMo, is not limited to particular task, we expect it to improve both the lemmatization and morphology tagging and achieve better MLAS and BLEX scores.

Implementation Details

We use the 100-dimensional pretrained word embeddings released by the shared task for the large languages. For the small treebanks and treebanks for low-resource languages where cross-lingual transfer is required, we use the 300-dimensional fasttext word embeddings. Old French treebank (fro_srcmf) presents the only exceptions and we use the French embeddings instead. For all the embeddings, we only use 10% of the most frequent words.

ELMo.

We use the same hyperparameter settings as Peters et al. (2018) for $\text{BiLSTM}^{(LM)}$ and the character CNN. We train their parameters as training a bidirectional language model on a set of 20-million-words data randomly sampled from the raw text released by the shared task for each language. Similar to Peters et al. (2018), we use the sample softmax technique to make training on large vocabulary feasible Jean et al. (2015). However, we use a window of 8192 words surrounding the target word as negative samples and it shows better performance in our preliminary experiments. The training of ELMo on one language takes roughly 3 days on an NVIDIA P100 GPU.

Biaffine Parser.

We use the same hyperparameter settings as Dozat et al. (2017). When trained with ELMo, we use a dropout of 33% on the projected vectors.

SCIR Tokenizer.

We use a 50-dimensional character bigram embeddings. For the character ELMo whose input is a character, the language model predict next character in the same way as the word ELMo. The final model is an ensemble of five single segmentors.

Uppsala Segmentor.

We use the default settings for the Uppsala segmentor and the final model is an ensemble of three single segmentors.

Results

We study the effect of ELMo on the large treebanks and report the results of a single tagger and parser with and without ELMo. Figure 1(a) shows the tagging results on the development set and Figure 1(b) shows the parsing results. Using ELMo in the tagger leads to a macro-averaged improvement of 0.56% in UPOS and the macro-averaged error reduction is 17.83%. Using ELMo in the parser leads to a macro-averaged improvement of 0.84% in LAS and the macro-averaged error reduction is 7.88%.

ELMo improves the tagging performance almost on every treebank, except for zh_gsd and gl_ctg. Similar trends are witnessed in the parsing experiments with ko_kaist and pl_lfg being the only treebanks where ELMo slightly worsens the performance.

We also study the relative improvements in dependence on the size of the treebank. The line in Figure 1(a) and Figure 1(b) shows the error reduction from using ELMo on each treebank. However, no clear relation is revealed between the treebank size and the gains using ELMo.

2 Effects of Ensemble

We also test the effect of ensemble and show the results in Figure 2. Parser ensemble leads to an averaged improvement of 0.55% in LAS and the averaged error reduction is 4.0%. These results indicate that ensemble is an effective way to improve the parsing performance. The relationship between gains using ensemble and treebank size is also studied in this figure and the trend is that small treebank benefit more from the ensemble. We address this to the fact that the ensemble improves the model’s generalization ability in which the parser trained on small treebank is weak due to overfitting.

3 Effects of Treebank Concatenation

As mentioned in Section 5, we study the effects of both the cross-domain concatenation and cross-lingual concatenation.

For the treebanks which have development set, the development performances are shown in Table 2. Numbers of sentences in the training set are also shown in this table. The general trend is that for the treebank with small training set, cross-domain concatenation achieves better performance. While for those with large training set, concatenation does not improve the performance or even worsen the results.

For the small treebanks which do not have development set, the 5 fold cross validation results are shown in Table 3 in which concatenation improves most of the treebanks except for gl_treegal.

Cross-Lingual Concatenation.

The experimental results of cross-lingual concatenation are shown in Table 4. Unfortunately, concatenating treebanks from different languages only achieves improved performance on uk_iu. This results also indicate that in cross lingual parsing, sophisticated methods like word embeddings transfer Guo et al. (2015, 2016b) and treebank transfer Guo et al. (2016a) are still necessary.

4 Effects of Better Preprocessing

We also study how preprocessing contributes to the final parsing performance. The experimental results on the development set are shown in Table 5. From this table, the performance of word segmentation is almost linearly correlated with the final performance. Similar trends on sentence segmentation performance are witnessed but el_gdt and pt_bosque presents some exceptions where better preprocess leads drop in the final parsing performance.

5 Parsing Strategies and Test Set Evaluation

Using the development set and cross validation, we choose the best model and data combination and the choices are shown in Table 6 along with the test evaluation. From this table, we can see that our system gains more improvements when both ELMo and parser ensemble are used. For some treebanks, concatenation also contributes to the improvements. Parsing Japanese, Vietnamese, and Chinese clearly benefits from better word segmentation. Since most of the participant teams use single parser for their system, we also remove the parser ensemble and do a post-contest evaluation. The results are also shown in this table. Our system without ensemble achieves an macro-averaged LAS of 75.26, which unofficially ranks the first according to LAS in the shared task.

We report the time and memory consumption. A full run over the 82 test sets on the TIRA virtual machine Potthast et al. (2014) takes about 40 hours and consumes about 4G RAM memory.

Conclusion

Our system submitted to the CoNLL 2018 shared task made several improvements on last year’s winning system from Dozat et al. (2017), including incorporating deep contextualized word embeddings, parser ensemble, and treebank concatenation. Experimental results on the development set show the effectiveness of our methods. Using these techniques, our system achieved an averaged LAS of 75.84% and obtained the first place in LAS in the final evaluation.

Credits

There are a few references we would like to give proper credit, especially to data providers: the core Universal Dependencies paper from LREC 2016 Nivre et al. (2016), the UD version 2.2 datasets Nivre et al. (2018), the baseline udpipe model released by Straka et al. (2016), the deep contextualized word embeddings code released by Peters et al. (2018), the biaffine tagger and parser released by Dozat et al. (2017), the joint sentence segmentor and tokenizer released by de Lhoneux et al. (2017), and the evaluation platform TIRA Potthast et al. (2014).

Acknowledgments

We thank the reviewers for their insightful comments, and the HIT-SCIR colleagues for the coordination on the machine usage. This work was supported by the National Key Basic Research Program of China via grant 2014CB340503 and the National Natural Science Foundation of China (NSFC) via grant 61300113 and 61632011.

References