A Comparison of Word Embeddings for the Biomedical Natural Language Processing

I Introduction

Word embeddings have been prevalently used in Natural Language Processing (NLP) applications due to the vector representations of words capturing useful semantic properties and linguistic relationships between words using deep neural networks . Word embeddings are commonly utilized as feature input to machine learning models, which enables machine learning techniques to process rew text data. There has been an increasing number of studies applying word embeddings in common NLP tasks, such as information extraction (IE) , information retrieval (IR) , sentiment analysis , question answering , and text summarization . Recently, in the biomedical domain, word embeddings have been remarkably utilized in applications such as biomedical named entity recognition (NER) , medical synonym extraction , relation extraction (RE) (e.g., chemical-disease relation , drug-drug interaction and protein-protein interaction ), biomedical IR and medical abbreviation disambiguation .

There are two main text resources utilized to train word embeddings for the biomedical NLP applications: internal task corpora (e.g., training data) and external data resources (e.g., Wikipedia) . The use of the former resource is straightforward as the internal corpora capture the nuances of language topic specific to the task . Exploiting external data resources is based on an implicit assumption that the external resources contain knowledge that could be used to enhance domain tasks . In addition, a number of pre-trained word embeddings are publicly available, such as the embeddings of Google Newshttps://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit and GloVehttps://nlp.stanford.edu/projects/glove/. These embeddings could capture the semantics of general English words from a large corpus. However, one question remains unanswered: do we need to train word embeddings for a specific NLP task given a number of public pre-trained word embeddings? This question becomes more significant for domain areas, and particularly more important for the clinical domain. The reason is that little electrical health record (EHR) data is publicly available due to the Health Insurance Portability and Accountability Act (HIPAA) requirements, while biomedical literature is more widely available through resources such as PubMedhttps://www.ncbi.nlm.nih.gov/pubmed/. However, to the best of our knowledge, there has been little work done to evaluate word embeddings trained from these textual resources for biomedical NLP applications.

In this study, we empirically evaluated word embeddings trained from four different corpora, namely clinical notes, biomedical publications, Wikipedia, and news. For the former two resources, we utilized clinical notes from the EHR system at Mayo Clinic and articles from PubMed Central (PMC)http://www.ncbi.nlm.nih.gov/pmc/tools/openftlist/ to train word embeddings. For the latter two resources, we used publicly available pre-trained word embeddings, GloVe and Google News. We performed the evaluation qualitatively and quantitatively. For the qualitative evaluation, we adopted the method used in Levy and Goldberg’s study , and manually inspected five of the most similar medical words to each of arbitrarily selected medical words from three medical categories (disorder, symptom, and drug). In addition, we analyzed word embeddings through a 2-dimensional visualization plot of 377 medical words. For the quantitative evaluation, we conducted both intrinsic and extrinsic evaluation. The intrinsic evaluation directly tested semantic relationships between medical words using four published datasets for measuring semantic similarity between medical terms, i.e., Pedersen , Hliaoutakis , MayoSRS , and UMNSRS . For the extrinsic evaluation, we applied word embeddings to downstream NLP applications in the biomedical domain including clinical IE, biomedical IR, and RE, and measured the performance of word embeddings.

II Related Work

Due to the success of word embeddings in a variety of NLP applications, some existing studies evaluate word embeddings in representing word semantics quantitatively. Most of them focus on evaluating the word embeddings generated by different approaches. Baroni et al. presented the first systematic evaluation of word embeddings generated by four models, i.e., DISSECThttp://clic.cimec.unitn.it/composes/, CBOW using word2vechttps://code.google.com/p/word2vec/, Distributional Memory modelhttp://clic.cimec.unitn.it/dm/, and 4) Collobert and Weston modelhttp://ronan.collobert.com/senna/ using a corpus of 2.8 billion tokens in the general English domain. They tested these models on fourteen benchmark datasets in five categories, including semantic relatedness, synonym detection, concept categorization, selectional preferences, and analogy. They found that the word2vec model, CBOW, performed the best for almost all the tasks.

Schnabel et al. trained the CBOW model of word2vec , C&W embeddings , Hellinger PCA , GloVe , TSCCA , and Sparse Random Projections on a 2008 GloVe dump, and tested on the same fourteen datasets. They found that the CBOW outperformed other embeddings on 10 datasets. They also conducted an extrinsic evaluation by using the embeddings as input features to two downstream tasks, namely noun phrase chunking and sentiment classification. They found the results of CBOW were also among the best.

Ghannay et al. conducted a similar intrinsic evaluation, they additionally evaluated the skip-gram models of word2vec , CSLM word embeddings , dependency-based word embeddings , and combined word embeddings on four NLP tasks, including Part-Of-Speech tagging, chunking, named entity recognition, mention detection, and two linguistic tasks. They trained these word embeddings on the Gigaword corpus composed of 4 billion words and found that the dependency-based word embeddings gave the best performance on the NLP tasks and that the combination of embeddings yielded significant improvement. Nayak et al’s study recommended that the evaluation of word embeddings should test both syntactic and semantic properties, and that the evaluation tasks should be closer to real-word applications. However, few of these studies evaluated word embeddings for tasks in the biomedical domain.

As most of the aforementioned studies evaluate word embeddings in the general (i.e., non-biomedical) NLP domain, only one recent study by Pakhomov et al. evaluates word embeddings in the biomedical domain, to the best of our knowledge. They trained the CBOW model on two biomedical corpora, namely clinical notes and biomedical publications, and one general English corpora, namely GloVe. The word embeddings were evaluated on subsets of UMNSRS dataset, which consisted of pairs of medical terms with the similarity of each pair assessed by medical experts, and on a document retrieval task and a word sense disambiguation task. They found that the semantics captured by the embeddings computed from biomedical publications were on par with that from clinical notes. We extended their evaluation of word embeddings by: 1) utilizing four datasets to evaluate word embeddings on capturing medical term semantics; 2) conducting a qualitative evaluation; and 3) examining word embeddings on more downstream applications with data provided by shared biomedical NLP tasks.

III Word Embeddings and Parameter Settings

We utilized word2vec in this study as it has been shown that word2vec generates better word embeddings for most general NLP tasks than other approaches . Since no evidence shows that the CBOW architecture outperforms the skip-gram architecture or vice versa, we arbitrarily chose the skip-gram architecture for word2vec.

where $J$ is the objective function, and $P(h|w)$ is the conditional probability in the neural probabilistic language model. $P(h|w)$ is usually defined by

where $\theta^{\prime}$ and $\theta$ are the input and output word embeddings, respectively. Accordingly, the log probability can be written as:

We can take the derivative of $J$ to obtain the embeddings, updating the equation iteratively. However, the computation is extremely expensive as in each iteration, the algorithm needs to go through the vocabulary $V$. By using negative-sampling, Mikolov et al. defined an empirical log probability $P^{\prime}(h|w)$ to approximate $P(h|w)$:

where $\sigma(x)=1/(1+\exp(-x))$ is a softmax function that normalizes a real vector into a probability vector, $P_{n}(h_{i})=\frac{f(h_{i})^{3/4}}{\sum_{i}^{|V|}f(h_{i})^{3/4}}$ is an empirical distribution that generates $k$ negative samples with $f(h_{i})$ being the term frequency for term $h_{i}$. The word embeddings $\theta$ can be computed by maximizing the objective function in Equation (1) by replacing $P(h|w)$ with $P^{\prime}(h|w)$.

We tested different vector dimensions of $D$ (i.e., 20, 60, 100) for the vector representation trained on EHR and MedLit and chose 100 for EHR and 60 for MedLit according to the performance in our intrinsic evaluation. Similarly, we chose the dimension of 100 for GloVe, and 300 for Google News since only 300 was publicly available for Google News. The experimental results of using different vector dimensions for the word embeddings are provided in Appendix A. For training word embeddings on the EHR and MedLit, we set the window size to 5, the minimum word frequency to 7 (i.e., the words that occurred less than 7 times in the corpus were ignored), and the negative sampling parameter to 5. These parameters were selected based on previous studies .

IV Data and Text Pre-prosessing

The first corpus, denoted as EHR, contains textual clinical notes for a cohort of 113k patients receiving their primary care at Mayo Clinic, spanning a period of 15 years from 1998 to 2013. The vocabulary size of this corpus is 103k. The second corpus, denoted as MedLit, is obtained from a snapshot of the Open Access Subsethttp://www.ncbi.nlm.nih.gov/pmc/tools/openftlist/ of PubMed Central (PMC)http://www.ncbi.nlm.nih.gov/pmc/ in March 2016, which is an online digital database of freely available full-text biomedical literature. It contains 1.25 million biomedical articles, and 2 million distinct words in the vocabulary. As a comparison, additional public pre-trained word embeddings from two general English resources, i.e., Google News https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp=sharing and GloVe http://nlp.stanford.edu/data/glove.6B.zip, were utilized in the evaluation. The Google News embeddings have vector representations for 3 million words from Google News, trained by the word2vec . The GloVe embeddings were trained by the GloVe model , and have 400k unique words in the vocabulary from a snapshot of Wikipedia in 2014 and Gigaword Fifth Editionhttps://catalog.ldc.upenn.edu/LDC2011T07.

The MedLit and EHR corpora were pre-processed minimally by removing punctuation, lowercasing, and replacing all digits with ”7”. One exception is that we replaced ‘-’ with ‘_’ if two or more words were connected by ‘-’ and treated these words as one. For the MedLit corpus, we additionally removed website urls, email addresses, and twitter handles. For the EHR corpus, the clinical narratives are written by medical practitioners, and thus contain more incomplete sentences than research articles. Therefore, we conducted additional pre-processing on the EHR corpus specific for the clinical notes from Mayo Clinic. Specifically, the section of “Family history” in the corpus was removed if it was semi-structured . As shown by an example in Table I, the semi-structured “Family history” section does not provide much valuable semantic information. The section of “Vital Signs” was totally removed since it did not contain contextual information for training word embeddings. Table II shows an example of the “Vital Signs” section in the EHR corpus. Moreover, we replaced all text contractions with their respective complete text (e.g., “can’t” is replaced with “can not”), and removed all the clinical notes metadata and note section headers, dates, phone numbers, weight and height information, and punctuation.

V Qualitative Evaluation

We arbitrarily selected medical words from three medical semantic categories, namely disorder, symptom, and drug. Word embeddings trained from four different corpora were utilized to compute the five most similar words to each selected medical word according to the cosine similarity. Then we adopted the method used in Levy and Goldberg’s study and manually inspected the conceptual similarity between the target word and the most similar words. Suppose $w_{1}$ and $w_{2}$ are two words, the similarity between $w_{1}$ and $w_{2}$ is defined as

where $\theta_{1}$ and $\theta_{2}$ are vector representations for $w_{1}$ and $w_{2}$ in the embedding space, respectively. If the target word is a medical phrase $s_{1}$ consisting of multiple words, i.e., $s_{1}={w_{1},w_{2},...,w_{n}}$, the similarity function becomes

where $\Theta_{1}=\frac{1}{n}\sum_{i}^{n}\theta_{i}$ is the representation for $s_{1}$ in the embedding space. This is different from Pakhomov et al’s study where only single word terms were considered. We ranked the words in the vocabulary based on the similarity to the target word and chose the five top ranked words.

Table III lists eight target words from the three medical categories, and the corresponding five most similar words computed by using the word embeddings trained from different resources.

For the first target word describing a disorder, diabetes, EHR and MedLit find its synonym, mellitus, in the most similar words while GloVe and Google News fail to find it. EHR finds two terms related to co-morbidities of diabetes, which are cholesterolemia and dyslipidemia, and a common adjective modifier term, uncontrolled. MedLit finds terms relevant to co-existing conditions for diabetes, such as cardiovascular (possibly from cardiovascular disease), nonalcoholic (possibly from nonalcoholic fatty liver disease), obesity, and polycystic (possibly from polycystic ovary syndrome which is a hyperandrogenic disorder that is associated with a high-risk of development of Type 2 diabetes). Most of these terms are related with medical research topics and occur frequently in the biomedical research articles. GloVe finds two related terms, hypertension and obesity, while three other terms, i.e., arthritis, cancer and alzheimer, are less relevant disease names. Google News finds two morphological terms, diabetics and diabetic, relevant to the target words, one synonym, diabetes_mellitus, and one related disease name, heart disease. We can draw similar conclusions for the second and third disorder words.

The dyspnea example in the symptom category demonstrates the advantage of EHR and MedLit. EHR finds palpitations, a common cause of dyspnea, and orthopnea, exertional, and doe (dyspnea on exertion) are synonyms or specific conditions for dyspnea. MedLit finds related symptoms, sweats and orthopnea, a synonym breathlessness, a relevant disorder hypotension, and a term relevant to the symptom rhonchi. GloVe finds synonyms shortness and breathlessness, and less relevant symptoms cyanosis and photophobia. Google News finds less relevant symptoms pruritus and rhinorrhea and less relevant disease nasopharyngitis. Similar observations can be found for sore throat and low blood pressure as well.

We can further observe that the semantics captured by the word embeddings trained from different corpora is disparate for the medical terms in the drug category. For opioid, EHR finds opiate, benzodiazepine, sedative, polypharmacy, which are very relevant medications. MedLit finds nmda_receptor, affective_motivational, naloxone_precipitated, hyperlocomotion, which are related to the mechanism of action of opioid. GloVe finds analgesic and less relevant anti-inflammatory, and Google News finds opioid-related phrases and relevant term antipsychotics. For the target term aspirin, EHR also finds very clinically relevant used terms and MedLit finds relevant terms in research articles while GloVe and Google News only find medication names.

It is obviously shown from these target words and the corresponding similar words that EHR and MedLit can capture the semantics of medical terms better than GloVe and Google News and find more relevant similar medical terms. However, EHR and MedLit find similar medical terms from different perspectives due to their focus difference. EHR contains clinical narratives and thus it is closer to clinical language. It contains terms with different morphologies and even typos, such as melitis, caner and thraot as listed in Table III. Differently, MedLit contains more medical terms used in research articles, and finds similar words mostly from a biomedical research perspective.

In order to show different aspects of medical concepts captured by word embeddings trained from different corpora, we extracted 377 medical terms from the UMNSRS dataset and visualized the word embeddings for these medical terms in a two-dimensional plot using t-distributed stochastic neighbor embedding (t-SNE) . Example clusters of medical terms in the word embeddings are shown in Figure 1. Figure 1a depicts a cluster of symptoms, such as heartburn, vomiting and nausea, from the word embeddings trained on EHR. Figure 1b shows a cluster of antibiotic medications, such as bacitracin, cefoxitin, and chloramphenicol, based on MedLit embeddings. Figures 1c and 1d illustrate clusters of symptoms from the GloVe and Google News embeddings, respectively. Since we did not employ any clustering method, these clusters were intuitively observed from the two-dimensional plot. The visualization of the entire set of 377 medical terms using word embeddings trained from four different corpora is provided in the supplementary file.

VI Quantitative Evaluation

We conducted both extrinsic and intrinsic quantitative evaluation, where the former used four published datasets for measuring semantic similarity between medical terms and the latter used downstream biomedical NLP tasks to evaluate word embeddings.

We tested word embeddings on four published biomedical measurement datasets commonly used to measure semantic similarity between medical terms. The first is Pedersen’s dataset that consists of 30 medical term pairs that were scored by physician experts according to their relatedness. The second is Hliaoutakis’s dataset consisting of 34 medical term pairs with similarity scores obtained by human judgments. The third, the MayoSRS dataset developed by Pakhomov et al. , consists of 101 clinical term pairs whose relatedness was determined by nine medical coders and three physicians from Mayo Clinic. The relatedness of each term pair was assessed based on a four point scale: (4.0) practically synonymous, (3.0) related, (2.0) marginally related and (1.0) unrelated. We evaluated the word embeddings using the mean score of the physicians and medical coders. The fourth, UMNSRS similarity dataset developed by Pakhomov et al. , consists of 566 medical term pairs whose semantic similarity was determined independently by eight medical residents from the University of Minnesota Medical School. The similarity and relatedness of each term pair was annotated based on a continuous scale by having the resident touch a bar on a touch sensitive computer screen to indicate the degree of similarity or relatedness.

For each pair of medical terms in the testing datasets, we used Equations (5) and (6) to calculate the semantic similarity for each pair. Since some medical terms may not exist in the vocabulary of word embeddings, we used fastText to compute word vectors for these out-of-vocabulary (OOV) medical terms. Specifically, we built character n-gram vectors analogous to fastText’s output by converting each word (e.g., “abcdef”) in the word embeddings to 3-gram (i.e., trigram) format (i.e., “abc”, “bcd”, “cde”, “def”) with vector representation of each trigram the same as that of the original word. After converting all the words, we utilized the averaged vector for the identical trigram extracted from different words (e.g., the vector for “abcdef” is $\theta_{1}$ and that for “defg” is $\theta_{2}$, the final vector for trigram “def” is $\frac{1}{2}(\theta_{1}+\theta_{2})$ since “def” is a shared trigram between the two words). Since each word with the number of characters greater than or equal to 3 can be represented as a bag of character trigrams, fastText represents an OOV medical term as the normalized sum of the vector representations of its trigrams . The Pearson correlation coefficient was employed to calculate the correlation between similarity scores from human judgments and those from word embeddings.

Table IV lists the Pearson correlation coefficient results for the four datasets. Overall, the semantic similarity captured by the word embeddings trained on EHR are closer to human experts’ judgments, compared with other word embeddings. MedLit performs worse than EHR but has a comparative result for the UMNSRS dataset. GloVe and Google News are inferior to EHR and MedLit, and perform similarly in representing medical semantics. Note that the four datasets and corresponding semantic similarity scores from both human experts and word embeddings are provided in the supplementary Excel file.

VI-B Extrinsic Evaluation

Extrinsic evaluations are used to measure the impact of word embeddings to specific biomedical NLP tasks. In this evaluation, we tested the word embeddings on three biomedical NLP tasks, namely clinical IE, biomedical IR, and RE.

Two clinical IE tasks were utilized to evaluate the word embeddings. The first task is an institutional task while the second is a shared task. Using the first task, we would like to examine whether the word embeddings trained on our institutional corpus perform better than external pre-trained word embeddings on a local institutional IE task. We also would like to investigate whether the results are consistent on a global shared task.

In the first experiment, we evaluated the word embeddings on an institutional IE task at Mayo Clinic. In this task, a set of 1000 radiology reports was given to detect whether a hand and figure/wrist fracture could be identified. Reports were drawn from a cohort of residents of Olmsted County, aged 18 or older, who experienced fractures in 2009-2011. Each report was annotated by a medical expert with multiple years of experience abstracting fractures by assigning “1” if a hand and figure/wrist fracture was found, or “0” otherwise.

In our experiment, the word embeddings were employed as features for machine learning models and evaluated by precision, recall, and F1 scores . For a clinical document $d=\{w_{1},w_{2},..,w_{M}\}$ where $w_{i},i=1,2,...,M$ is the $i$th word and $M$ is the total number of words in this document, the feature vector $\mathbf{x}$ of document $d$ is defined by

where $\mathbf{x}_{i}$ is the embedding vector for word $w_{i}$ from the word embedding matrix. Then $\mathbf{x}$ was utilized as input to a conventional machine learning model, which is Support Vector Machine (SVM) in this experiment. We performed 10-fold cross validation on the dataset. The means of precision, recall, and F1 scores from the 10-fold cross validation was reported, which are defined below:

where TP, TN, FP, and FN represent true positives, true negatives, false positives, and false negatives, respectively, and $i={1,2,...,10}$ represents the $i$th fold cross validation. As a comparison, the baseline method used term frequency features as input.

The experimental results are listed in Table V. The word embeddings trained on EHR are superior to other word embeddings in terms of all metrics (precision: 0.974, recall: 0.972, F1 score: 0.972) with statistical significance using t-test (p$<$0.01). The fracture dataset in this experiment is curated from the same EHR system as the EHR corpus used to train word embeddings, and thus they have identical sublanguage characteristics. The word embeddings trained on MedLit also have comparable results (precision: 0.946, recall: 0.943, F1 score: 0.942). Since this task is a medical task with specific medical terminologies, the word embeddings trained on Google News have the worst performance. However, the word embeddings trained on GloVe are close to those trained on EHR with 0.02 difference on F1 score without statistical significance (p$<$0.01). This experiment shows that word embeddings trained on a local corpus have the best performance for a local task but those trained on an external Wikipedia corpus also have comparable performance.

Secondly, we tested the word embeddings on the 2006 i2b2 (Informatics for Integrating Biology to the Bedside) smoking status extraction shared task . Participants of this task were asked to develop automatic NLP systems to determine the smoking status of patients from their discharge records in Partners HealthCare. For each discharge record, an automatic system should be able to categorize it into five pre-determined smoking status categories: past smoker, current smoker, smoker, non-smoker, and unknown, where a past and a current smoker are distinguished based on temporal expressions in the patient’s medical records. The dataset contains a total of 389 documents, including 35 documents of current smoker, 66 of non-smoker, 36 of past smoker, and 252 of unknown. The settings of this shared task are identical to those of the previous local institutional IE task: SVM was utilized as the machine learning model; 10-fold cross validation was performed; term frequency features were used as input in the baseline; and the means of precision, recall and F1 scores were obtained as metrics.

The experimental results are shown in Table VI. First, it is obvious that the word embedding features perform better than term frequency features due to the semantics embedded in word embeddings, which is consistent with the previous local institutional IE task. The word embeddings trained on EHR produced the best performance with a F1 score of 0.900. The reason might be that the smoking dataset has the similar sublanguage characteristics as the EHR corpus. This result indicates that the effective word embeddings can be shared across institutions for clinical IE tasks. Another interesting observation is that the performance of word embeddings trained on Google News is close to that trained on EHR corpus with a comparable F1 score and a better recall. The performance difference is not statistically significant (p$<$0.01). This implies that word embeddings trained on a public dataset may not be definitely inferior to these trained on a medically specific dataset for a medical IE task. The likely cause is that the terminology used in the smoking status extraction task also appears frequently in the news, such as medications and advice for smokers.

VI-B2 Biomedical Information Retrieval

To evaluate word embeddings for biomedical IR, we utilized the dataset provided by the Text REtreival Conference 2016 Clinical Decision Support (TREC 2016 CDS) track. The TREC 2016 CDS track focuses on biomedical literature retrieval that helps physicians find the precise literature information and make the best clinical decision at the point of care . The query topics were generated from EHRs in the MIMIC-III dataset . Those topics were categorized into three most common types, Diagnosis, Test and Treatment, according to physicians’ information needs, and 10 topics were provided for each type. Each topic is comprised of a note field (admission note), a description field (jargons and clinical abbreviations are removed) and a summary field (simplified version of the description). The participants were required to use only one of these three fields in their submissions and at least one submission must utilize the note field. Submitted systems should retrieve relevant biomedical articles from a given PMC article collection for each given query topic to answer three corresponding clinical questions: What is the patient’s diagnosis? What tests should the patient receive? How should the patient be treated?. Each IR system can retrieve up to 1000 documents per query.

In order to make the comparison as fair as possible, we first implemented a simple IR system as the baseline system using the original queries following the study in , and then employed the simplest query expansion method using the word embeddings. We used the summary field in the query, removed the stop words, and expanded each of the left query terms with five most similar terms from word embeddings. Take the query “A 78 year old male presents with stools and melena” as an example, the term “male” was expanded by “female gentleman lady caucasian man”, “stools” by “stooling liquidy voluminous semiformed tenesmus”, and “melena” by “hematemesis hematochezia melana brbpr hematemasis”. We assigned weight 0.8 to the original query and 0.2 to the expanded query. Indri was utilized as our indexing and retrieval tool. The preprocessing for the corpus included stopword removal and Porter stemming. The stopword list was based on the MedLit stopwords http://www.ncbi.nlm.nih.gov/books/NBK3827/table/pubmedhelp.T.stopwords/. The article-id, title, abstract, and body fields of each document in the corpus were indexed. Language models with two-stage smoothing was used to obtain all the retrieval results. Four official metrics, namely Inferred Normalized Discounted Cumulated Gain (infNDCG), Inferred Average Precision (infAP), Precision at 10 (P@10), and Mean Average Precision (MAP), were utilized to measure the IR performance. infNDCG measures the document ranking quality of an IR system; infAP measures the retrieval effectiveness given incomplete judgments for an IR system; P@10 is the number of relevance documents among the top 10; and MAP is the mean of the average precision scores for each query in a set of queries.

Table VII lists the results of using the word embeddings trained from different resources for query expansion on the TREC 2016 CDS track. It is interesting that the word embeddings based query expansion method failed to improve the retrieval performance, and even worsened the performance when infAP and MAP were metrics. By comparing the retrieval performance, we observe that EHR and MedLit perform slight better than GloVe and Google News without statistical significance (p$<$0.01). This result implies that applying word embeddings trained from different resources has no significant improvement for the biomedical IR task.

VI-B3 Relation Extraction

For the RE task, we considered drug-drug interaction (DDI) extraction, which is a specific RE task in the biomedical domain. DDI is an unexpected change in a drug’s effect on the human body when the drug and a second drug are co-prescribed and taken together. Automatically extracting DDI information from literature is a challenging and important research topic since the volume of the published literature grows rapidly and greatly. In this experiment, we evaluated the word embeddings on the DDIExtraction 2013 challenge corpus . The dataset for DDIExtraction 2013 was composed of sentences describing DDIs from the DrugBank database and MedLine abstracts. In this dataset, drug entities and DDIs were annotated at the sentence level and each sentence could contain two or more drugs. An RE system should be able to automatically extract DDI drug pairs from a sentence. We exploited the baseline system introduced in where features include words and word bigrams with binary values indicating their presence or absence in a sentence, cosine similarity between centroid vector of each class and the instance, negation (three features indicating negation before the first main drug, between two main drugs, and after the two main drugs). We concatenated the word embeddings to the baseline features and tested the performance. Since Random Forest has the best performance in , we utilized it as the classifier with 10-fold cross validation.

Table VIII shows the F1 scores of Random Forest using word embeddings trained from different resourceson the DDIExtraction 2013 challenge. We can see that the overall performance of word embeddings trained on Google News is the best. The reason is that the semantics of general English terms in the context of drug mentions are more important for determining the drug interactions. For example, in the sentence “Acarbose may interact with metformin”, the term “interact” is crucial to classify the relation. Since these crucial terms are generally not medical terminology, word embeddings trained on Google News where the corpus represents general English are able to capture the semantics of these terms. However, Google News outperformed other resources but not conclusive in statistical significance using t-test (p$<$0.01). Another interesting observation is that word embeddings trained from MedLit have the best performance for the DrugBank corpus while these from Google News perform the best for the MedLine corpus. Though MedLine abstracts are from MedLit articles, this result shows that word embeddings trained from the same corpus are not necessarily superior to other embeddings.

VII Conclusion and Discussion

In this study, we provide an empirical evaluation of word embeddings trained from four different corpora, namely clinical notes, biomedical publications, Wikipedia, and news. We performed the evaluation qualitatively and quantitatively. For the qualitative evaluation, we selected a set of medical words and impressionistically evaluated the five most similar medical words. We then analyzed word embeddings through the visualization of those word embeddings. We conducted both extrinsic and intrinsic evaluation for the quantitative evaluation. The intrinsic evaluation directly tested semantic relationships between medical words using four published datasets for measuring semantic similarity between medical terms while the extrinsic evaluation evaluated word embeddings in three downstream biomedical NLP applications, i.e., clinical IE, biomedical IR, and RE.

Based on the evaluation results, we can draw the following conclusions. First, the word embeddings trained on EHR and MedLit can capture the semantics of medical terms better than those trained on GloVe and Google News, and find more relevant similar medical terms. However, EHR finds similar terms vis a vis clinical language while MedLit contains more medical terminology used in medical articles, and finds similar words mostly from a medical research perspective. Second, the medical semantic similarity captured by the word embeddings trained on EHR and MedLit are closer to human experts’ judgments, compared to these trained on GloVe and Google News. Third, there does not exist a consistent global ranking of word embeddings for the downstream biomedical NLP applications. However, adding word embeddings as extra features will improve results on most downstream tasks. Finally, word embeddings trained from biomedical domain corpora do not necessarily have better performance than those trained on other general domain corpora. That is, there might be no significant difference when word embeddings trained from an out-domain corpus are employed for a biomedical NLP application. However, the performance of word embeddings trained from a local institutional corpus might perform better for local institutional NLP tasks.

Our experiments implicitly show that applying word embeddings trained from corpora in a general domain, such as Wikipedia and news, is not significantly inferior to applying those obtained from biomedical or clinical domain, which is usually difficult to access due to privacy. This result is consistent with but more general than the conclusion drawn in . Thus, a lack of access to a domain-specific corpus is not necessarily a barrier for the use of word embeddings in practical implementations.

As a future direction, we would like to evaluate word embeddings on more downstream biomedical NLP applications, such as medial named entity recognition and clinical note summarization. We will investigate whether word embeddings trained from different resources represent language characteristics differently for a corpus, such as term frequency and medical concepts. We also want to assess word embeddings across health care institutions using different EHR systems and investigate how sublanguage characteristics affect the portability of word embeddings. Moreover, we want to apply clustering methods on word embeddings and compare the word-level and concept-level difference between clusters of medical terms.

There are a few limitations in this study. First, we only examined the word embeddings trained on the EHR from Mayo Clinic, which might have introduced bias into the conclusion as the EHR quality may vary by institutions. However, it is challenging to obtain word embeddings trained on EHR data from multiple sites. We are currently exploring the use of privacy-preserving techniques for obtaining embeddings from multiple sites leveraging our prior work to have more generalizable embeddings. Second, we tested only two widely used public pre-trained word embeddings. There are a number of word embeddings publicly availablehttps://github.com/3Top/word2vec-api. Third, the generalizability of the results for the biomedical IR and RE tasks may be questionable since we only used one shared task dataset for each task to evaluate the word embeddings.

VIII Acknowledgement

This work has been supported by the National Institute of Health (NIH) grants R01LM011934, R01GM102282, and U01TR002062.

References

Appendix A Intrinsic evaluation of word embeddings with different dimensions.

Table IX shows the intrinsic evaluation results of word embeddings using different dimensions.