De-identification of Patient Notes with Recurrent Neural Networks

Introduction and related work

In many countries such as the United States, medical professionals are strongly encouraged to adopt electronic health records (EHRs) and may face financial penalties if they fail to do so [DesRoches et al., 2013, Wright et al., 2013]. The Centers for Medicare & Medicaid Services have paid out more than $30 billion in EHR incentive payments to hospitals and providers who have attested to meaningful use as of March 2015. Medical investigations may greatly benefit from the resulting increasingly large EHR datasets. One of the key components of EHRs is patient notes: the information they contain can be critical for a medical investigation because much information present in texts cannot be found in the other elements of the EHR. However, before patient notes can be shared with medical investigators, some types of information, referred to as protected health information (PHI), must be removed in order to preserve patient confidentiality. In the United States, the Health Insurance Portability and Accountability Act (HIPAA) [Office for Civil Rights, 2002] defines 18 different types of PHI, ranging from patient names to phone numbers. Table 1 presents the exhaustive list of PHI types as defined by HIPAA.

The task of removing PHI from a patient note is referred to as de-identification, since the patient cannot be identified once PHI is removed. De-identification can be either manual or automated. Manual de-identification means that the PHI are labeled by human annotators. There are three main shortcomings of this approach. First, only a restricted set of individuals is allowed to access the identified patient notes, thus the task cannot be crowdsourced. Second, humans are prone to mistakes. [Neamatullah et al., 2008] asked 14 clinicians to detect PHI in approximately 130 patient notes: the results of the manual de-identification varied from clinician to clinician, with recall ranging from 0.63 to 0.94. [Douglass et al., 2005, Douglas et al., 2004] reported that annotators were paid US$50 per hour and read 20,000 words per hour at best. As a matter of comparison, the MIMIC dataset [Goldberger et al., 2000, Saeed et al., 2011], which contains data from 50,000 intensive care unit (ICU) stays, consists of 100 million words. This would require 5,000 hours of annotation, which would cost US$250,000 at the same pay rate. Given the annotators’ spotty performance, each patient note would have to be annotated by at least two different annotators, so it would cost at least US$500,000 to de-identify the notes in the MIMIC dataset.

Automated de-identification systems can be classified into two categories: rule-based systems and machine-learning-based systems. Rule-based systems typically rely on patterns, expressed as regular expressions and gazetteers, defined and tuned by humans. They do not require any labeled data (aside from labels required for evaluating the system), and are easy to implement, interpret, maintain, and improve, which explains their large presence in the industry [Chiticariu et al., 2013]. However, they need to be meticulously fine-tuned for each new dataset, are not robust to language changes (e.g., variations in word forms, typographical errors, or infrequently used abbreviations), and cannot easily take into account the context (e.g., “Mr. Parkinson” is PHI, while “Parkinson’s disease” is not PHI). Rule-based systems are described in [Berman, 2003, Beckwith et al., 2006, Fielstein et al., 2004, Friedlin and McDonald, 2008, Gupta et al., 2004, Morrison et al., 2009, Neamatullah et al., 2008, Ruch et al., 2000, Sweeney, 1996, Thomas et al., 2002].

To alleviate some downsides of the rule-based systems, there have been many attempts to use supervised machine learning algorithms to de-identify patient notes by training a classifier to label each word as PHI or not PHI, sometimes distinguishing between different PHI types. Common statistical methods include decision trees [Szarvas et al., 2006], log-linear models, support vector machines [Guo et al., 2006, Uzuner et al., 2008, Hara, 2006], and conditional random fields [Aberdeen et al., 2010], the latter being employed in most of the state-of-the-art systems. For a thorough review of existing systems, see [Meystre et al., 2010, Stubbs et al., 2015]. All these methods share two downsides: they require a decent sized labeled dataset and much feature engineering. As with rules, quality features are challenging and time-consuming to develop.

Recent approaches to natural language processing based on artificial neural networks (ANNs) do not require handcrafted rules or features, as they can automatically learn effective features by performing composition over tokens which are represented as vectors, often called token embeddings. The token embeddings are jointly learned with the other parameters of the ANN. They can be initialized randomly, but can be pre-trained using large unlabeled datasets typically based on token co-occurrences [Mikolov et al., 2013b, Collobert et al., 2011, Pennington et al., 2014]. The latter often performs better, since the pre-trained token embeddings explicitly encode many linguistic regularities and patterns. As a result, methods based on ANNs have shown promising results for various tasks in natural language processing, such as language modeling [Mikolov et al., 2010], text classification [Socher et al., 2013, Kim, 2014, Blunsom et al., 2014, Lee and Dernoncourt, 2016], question answering [Weston et al., 2015, Wang and Nyberg, 2015], machine translation [Bahdanau et al., 2014, Tamura et al., 2014, Sundermeyer et al., 2014], as well as named entity recognition [Collobert et al., 2011, Lample et al., 2016, Labeau et al., 2015]. A few methods also use vector representations of characters as inputs in order to either replace or augment token embeddings [Kim et al., 2015, Lample et al., 2016, Labeau et al., 2015].

Inspired by the performance of ANNs for various other NLP tasks, this article introduces the first de-identification system based on ANNs. Unlike other machine learning based systems, ANNs do not require manually-curated features, such as those based on regular expressions and gazetteers. We show that ANNs achieve state-of-the-art results on de-identification of two different datasets for patient notes, the i2b2 2014 challenge dataset and the MIMIC dataset.

Methods and materials

We first present a de-identifier we developed based on a conditional random field (CRF) model in Section 2.1. This de-identifier yields state-of-the-art results on the i2b2 2014 dataset, which is the reference dataset for comparing de-identification systems. This system will be used as a challenging baseline for the ANN model that we will present in Section 2.2. The ANN model outperforms the CRF model, as outlined in Section 3.

In the CRF model, each patient note is tokenized and features are extracted for each token. During the training phase, the CRF’s parameters are optimized to maximize the likelihood of the gold standard labels. During the test phase, the CRF predicts the labels. The performance of a CRF model depends mostly on the quality of its features. We used a combination of n-gram, morphological, orthographic, and gazetteer features. These are similar to features used in the best-performing CRF-based competitors in the i2b2 challenge [Yang and Garibaldi, 2015, Liu et al., 2015].

In order to effectively incorporate context when predicting a label, the features for a given token are computed based on that token and on the four surrounding tokens.

2 ANN model

The main components of the ANN model are recurrent neural networks (RNNs). In particular, we use a type of RNN called Long Short Term Memory (LSTM) [Hochreiter and Schmidhuber, 1997], as discussed in Section 2.2.1.

Character-enhanced token embedding layer (Section 2.2.2),

Label sequence optimization layer (Section 2.2.4).

The character-enhanced token embedding layer maps each token into a vector representation. The sequence of vector representations corresponding to a sequence of tokens are input to the label prediction layer, which outputs the sequence of vectors containing the probability of each label for each corresponding token. Lastly, the sequence optimization layer outputs the most likely sequence of predicted labels based on the sequence of probability vectors from the previous layer. All layers are learned jointly. Figure 1 shows the ANN architecture.

In the following, we denote scalars in italic lowercase (e.g., $k$, $b_{f}$), vectors in bold lowercase (e.g., $\mathbf{s},\,\mathbf{x}_{i}$), and matrices in italic uppercase (e.g., $W_{f}$) symbols. We use the colon notations $x_{i:j}$ and $\mathbf{v}_{i:j}$ to denote the sequence of scalars $(x_{i},\dotsc,x_{j})$, and vectors $(\mathbf{v}_{i},\mathbf{v}_{i+1},\dotsc,\mathbf{v}_{j})$, respectively.

RNN is a neural network architecture designed to handle input sequences of variable sizes, but it fails to model long term dependencies. LSTM is a type of RNN that mitigates this issue by keeping a memory cell that serves as a summary of the preceding elements of an input sequence. More specifically, given a sequence of vectors $\mathbf{x}_{1},\mathbf{x}_{2},\dotsc,\mathbf{x}_{n},$ at each step $t=1,\dotsc,n$, an LSTM takes as input $\mathbf{x}_{t},\mathbf{h}_{t-1},\mathbf{c}_{t-1}$ and produces the hidden state $\mathbf{h}_{t}$ and the memory cell $\mathbf{c}_{t}$ based on the following formulas:

where $W_{i},W_{c},W_{o}$ are weight matrices and $\mathbf{b}_{i},\mathbf{b}_{c},\mathbf{b}_{o}$ are bias vectors used in the input gate, memory cell, and output gate calculations, respectively. The symbols $\sigma(\cdot)$ and tanh$(\cdot)$ refer to the element-wise sigmoid and hyperbolic tangent functions, and $\odot$ is the element-wise multiplication. $\mathbf{h}_{0}=\mathbf{c}_{0}=\mathbf{0}$.

A bidirectional LSTM consists of a forward LSTM and a backward LSTM, where the forward LSTM calculates the forward hidden states $(\overrightarrow{\mathbf{h}}_{1},\overrightarrow{\mathbf{h}}_{2},\dotsc,\overrightarrow{\mathbf{h}}_{n}),$ and the backward LSTM calculates the backward hidden states $(\overleftarrow{\mathbf{h}}_{1},\overleftarrow{\mathbf{h}}_{2},\dotsc,\overleftarrow{\mathbf{h}}_{n})$ by feeding the input sequence in the backward order, from $\mathbf{x}_{n}$ to $\mathbf{x}_{1}.$

Depending on the application of the LSTM, one might need an output sequence corresponding to each element in the sequence, or a single output that summarizes the whole sequence. In the former case, the output sequence $\mathbf{h}_{1},\mathbf{h}_{2},\dotsc,\mathbf{h}_{n}$ of the LSTM is obtained by concatenating the hidden states of the forward and the backward LSTMs for each element i.e., $\overleftrightarrow{\mathbf{h}_{t}}=(\overrightarrow{\mathbf{h}}_{t};\overleftarrow{\mathbf{h}}_{t})$ for $t=1,\dotsc,n.$ In the latter case, the output is obtained by concatenating the last hidden states of the forward and the backward LSTMs i.e., $\overleftrightarrow{\mathbf{h}}=(\overrightarrow{\mathbf{h}}_{n};\overleftarrow{\mathbf{h}}_{n}).$

2.2 Character-enhanced token embedding layer

The character-enhanced token embedding layer takes a token as input and outputs its vector representation. The latter results from the concatenation of two different types of embeddings: the first one directly maps a token to a vector, while the second one comes from the output of a character-level token encoder.

The direct mapping $\mathcal{V}_{T}(\cdot)$ from token to vector, often called a token (or word) embedding, can be pre-trained on large unlabeled datasets using programs such as word2vec [Mikolov et al., 2013b, Mikolov et al., 2013a, Mikolov et al., 2013c] or GloVe [Pennington et al., 2014], and can be learned jointly with the rest of the model. Token embeddings, often learned by sampling token co-occurrence distributions, have desirable properties such as locating semantically similar words closely in the vector space, hence leading to state-of-the-art performance for various tasks.

While the token embeddings capture the semantics of tokens to some degree, they may still suffer from data sparsity. For example, they cannot account for out-of-vocabulary tokens, misspellings, and different noun forms or verb endings. One solution to remediate some of these issues would be to lemmatize tokens before training, but this approach may fail to retain some useful information such as the distinction between some verb and noun forms.

We address this issue by using character-based token embeddings, which incorporate each individual character of a token to generate its vector representation. This approach enables the model to learn sub-token patterns such as morphemes (e.g., suffix or prefix) and roots, thereby capturing out-of-vocabulary tokens, different surface forms, and other information not contained in the token embeddings.

As a result, the final output $\mathbf{e}_{i}$ of the character-enhanced token embedding layer for $i^{th}$ token $x_{i}$ is the concatenation of the token embedding $\mathcal{V}_{T}(x_{i})$ and the character-based token embedding $\overleftrightarrow{\mathbf{b}_{i}}.$ In summary, when the character-enhanced token embedding layer receives a sequence of tokens $x_{1:n}$ as input, it will output the sequence of token embeddings $\mathbf{e}_{1:n}$.

2.3 Label prediction layer

The label prediction layer takes as input the sequence of vectors $\mathbf{e}_{1:n}$, i.e., the outputs of the character-enhanced token embedding layer, and outputs $\mathbf{a}_{1:n}$, where the $t^{th}$ element of $\mathbf{a}_{n}$ is the probability that the $n^{th}$ token has the label $t$. The labels are either one of the PHI types or non-PHI. For example, if one aims to predict all 18 HIPAA-defined PHI types, there would be 19 different labels.

The label prediction layer contains a bidirectional LSTM that takes the input sequence $\mathbf{e}_{1:n}$ and generates the corresponding output sequence $\overleftrightarrow{\mathbf{d}_{1:n}}.$ Each output $\overleftrightarrow{\mathbf{d}_{i}}$ of the LSTM is given to a feed-forward neural network with one hidden layer, which outputs the corresponding probability vector $\mathbf{a}_{i}$.

2.4 Label sequence optimization layer

The label sequence optimization layer takes the sequence of probability vectors $\mathbf{a}_{1:n}$ from the label prediction layer as input, and outputs a sequence of labels $y_{1:n}$, where $y_{i}$ is the label assigned to the token $t_{i}$.

The simplest strategy to select the label $y_{i}$ would be to choose the label that has the highest probability in $\mathbf{a}_{i}$, i.e. $y_{i}=\operatorname{argmax}_{k}{\mathbf{a}_{i}[k]}$. However, this greedy approach fails to take into account the dependencies between subsequent labels. For example, it may be more likely to have a token with the PHI type STATE followed by a token with the PHI type ZIP than any other PHI type. Even though the label prediction layer has the capacity to capture such dependencies to a certain degree, it may be preferable to allow the model to directly learn these dependencies in the last layer of the model.

One way to model such dependencies is to incorporate a matrix $T$ that contains the transition probabilities between two subsequent labels. $T[i,j]$ is the probability that a token with label $i$ is followed by a token with the label $j$. The score of a label sequence $y_{1:n}$ is defined as the sum of the probabilities of individual labels and the transition probabilities:

These scores can be turned into probabilities of the label sequences by taking a softmax function over all possible label sequences. During the training phase, the objective is to maximize the log probability of the gold label sequence. In the testing phase, given an input sequence of tokens, the corresponding sequence of predicted labels is chosen as the one that maximizes the score.

Experiments and results

We evaluate our two models on two datasets: i2b2 2014 and MIMIC de-identification datasets. The i2b2 2014 dataset was released as part of the 2014 i2b2/UTHealth shared task Track 1 [Stubbs et al., 2015]. It is the largest publicly available dataset for de-identification. Ten teams participated in this shared task, and 22 systems were submitted. As a result, we used the i2b2 2014 dataset to compare our models against state-of-the-art systems.

The MIMIC de-identification dataset was created for this work as follows. The MIMIC-III dataset [Johnson et al., 2016, Goldberger et al., 2000, Saeed et al., 2011] contains data for 61,532 ICU stays over 58,976 hospital admissions for 46,520 patients, including 2 million patient notes. In order to make the notes publicly available, a rule-based de-identification system [Douglass, 2005, Douglass et al., 2005, Douglas et al., 2004] was written for the specific purpose of de-identifying patient notes in MIMIC, leveraging dataset-specific information such as the list of patient names or addresses. The system favors recall over precision: there are virtually no false negatives, while there are numerous false positives. To create the gold standard MIMIC de-identification dataset, we selected 1,635 discharge summaries, each belonging to a different patient, containing a total of 60.7k PHI instances. We then annotated the PHI instances detected by the rule-based system as true positives or false positives. We found that 15% of the PHI instances detected by the rule-based system were false positives.

Table 1 introduces the PHI types and Table 2 presents the datasets’ sizes. For the test set, we used the official test set for the i2b2 dataset, which is 40% of the dataset; we randomly selected 20% of the MIMIC dataset as the test set for this dataset.

2 Evaluation metrics

To assess the performance of the two models, we computed the precision, recall, and F1-score. Let TP be the number of true positives, FP the number of false positives, and FN the number of false negatives. Precision, recall, and F1-score are defined as follows: $\text{precision}=\frac{TP}{TP+FP}$, $\text{recall}=\frac{TP}{TP+FN}$, and $\text{F1-score}=\frac{2*\text{precision}*\text{recall}}{\text{precision}+\text{recall}}$. Intuitively, precision is the proportion of the predicted PHI labels that are gold labels, recall is the proportion of the gold PHI labels that are correctly predicted, and F1-score is the harmonic mean of precision and recall.

3 Training and hyperparameters

The model is trained using stochastic gradient descent, updating all parameters, i.e., token embeddings, character embeddings, parameters of bidirectional LSTMs, and transition probabilities, at each gradient step. For regularization, dropout is applied to the character-enhanced token embeddings before the label prediction layer. Below are the choices of hyperparameters and token embeddings, optimized using a subset of the training set:

character-based token embedding LSTM dimension: 25

We tried pre-training token embeddings on the i2b2 2014 dataset and the MIMIC datasetFor MIMIC, we used the entire dataset containing 2 million notes and 800 million tokens. using word2vec and GloVe. Both word2vec and GloVe were trained using a window size of 10, a minimum vocabulary count of 5, and 15 iterations. Additional parameters of word2vec were the negative sampling and the model type, which were set to 10 and skip-gram, respectively. We also experimented with the publicly availablehttp://nlp.stanford.edu/projects/glove/ token embeddings such as GloVe trained on Wikipedia and Gigaword 5 [Parker et al., 2011]. The results were quite robust to the choice of the pre-trained token embeddings. The GloVe embeddings trained on Wikipedia articles yielded slightly better results, and we chose them for the rest of this work.

4 Results

All results were computed using the official evaluation script from the i2b2 2014 de-identification challenge. Table 3 presents the main results, based on binary token-based precision, recall, and F1-score for HIPAA-defined PHI only. These PHI types are the most important since only those are required to be removed by law. On the i2b2 dataset, our ANN model has a higher F1-score and recall than our CRF model as well as the best system from the i2b2 2014 de-identification challenge, which was the Nottingham system [Yang and Garibaldi, 2015]. The only freely available, off-the-shelf program for de-identification, called the MITRE Identification Scrubber Toolkit (MIST) [Aberdeen et al., 2010], performed poorly. Combining the outputs of our ANN and CRF models, by considering a token to be PHI if it is identified as such by either model, further increases the performance in terms of F1-score and recall.

It should be noted that the Nottingham system was specifically fine-tuned for the i2b2 dataset as well as the i2b2 evaluation script. For example, the Nottingham system post-processes the detected PHI terms in order to match the offset of the gold PHI tokens, such as modifying “MR:6746781” to “6746782” and “MWFS” to “M”, “W”, “F”, “S”.

On the MIMIC dataset, our ANN model also has a higher F1-score and recall than our CRF model. Interestingly, combining the outputs of our ANN and CRF models did not increase the F1-score, because precision was negatively impacted. However, the recall did benefit from combining the two models. MIST was much more competitive on this dataset.

We calculated the statistical significance of the differences in precision, recall, and F1-score between the CRF and ANN models using approximate randomization with 9999 shuffles. The significance levels of the differences in precision, recall, and F1-score are 0.37, 0.02, 0.22 for the i2b2 dataset, and 0.08, 0.00, 0.00 for the MIMIC dataset, respectively.

5 Error analysis

Figure 2 shows the binary token-based F1-scores for each PHI category. The ANN model outperforms the CRF model on all categories for both datasets, with the exception of the ID (which mostly contains medical record numbers) category in the i2b2 dataset. This is due to the fact that the CRF model uses sophisticated regular expression features that are tailored to detect ID patterns such as “38:Z8912708G”.

Another interesting difference between the ANN and the CRF results is the PROFESSION category: the ANN significantly outperforms the CRF. The reason behind this result is that the embeddings of the tokens that represent a profession tend to be close in the token embedding space, which allows the ANN model to generalize well. We tried assembling various gazetteers for the PROFESSION category, but all of them were performing significantly worse than the ANN model.

Table 4 presents some examples of gold PHI instances correctly predicted by the ANN model that the CRF model failed to predict, and conversely. This illustrates that the ANN model efficiently copes with the diversity of the contexts in which tokens appear, whereas the CRF model can only address the contexts that are manually encoded as features. In other words, the ANN model’s intrinsic flexibility allows it to better capture the variance in human languages than the CRF model. For example, it would be challenging and time-consuming to engineer features for all possible contexts such as “had a stroke at 80”, “quit smoking in 08”, “on the 29th of this month”, and “his friend Epstein”. The ANN model is also very robust to variations in surface forms, such as misspellings (e.g., “in teh late 60s”, “Khazakhstani”, “01/19/:0”), tokenizations (e.g., “Results02/20/2087”, “MC # 0937884Date”), and different phrases referring to the same semantic meaning (e.g., “San Rafael Mount Hospital”, “Rafael Mount”, “Rafael Hospital”). Furthermore, the ANN model is able to detect many PHI instances despite not having explicit gazetteers, as examples in the LOCATION and PROFESSION categories illustrate. We conjecture that the character-enhanced token embeddings contain rich enough information to effectively function as gazetteers, as tokens with similar semantics are closely located in the vector representation [Mikolov et al., 2013b, Collobert et al., 2011, Kim et al., 2015].

On the other hand, CRF is good at rarely occurring patterns that are written in highly specialized regular expression patterns (e.g., “38:Z8912708G”, “53RHM”) or tokens that are included in the gazetteers (e.g., “Christmas”, “WPH”, “rosenberg”, “Motor Vehicle Body Repairer”). For example, the PHI token “Christmas” only occurs in the test set, and unless the context gives a strong indication, the ANN model cannot detect it, whereas the CRF model could, as long as it is included in the gazetteers.

6 Effect of training set size

Figure 3 shows the impact of the training set size on the performance of the models on the MIMIC dataset. When the training set size is very limited, the CRF performs slightly better than the ANN model, since the CRF model can leverage handcrafted features without much training data. As the training set size increases, the ANN model starts to significantly outperform the CRF model, since the parameters including the embeddings are automatically fine-tuned with more data, and therefore the features learned by the ANN model become increasingly more refined than the manually handcrafted features. As a result, combining the outputs of the CRF and ANN models increases the F1-score over the ANN model only for small training set size and yields a less competitive F1-score than the ANN model for bigger training set size.

7 Ablation analysis

In order to quantify the importance of various elements of the ANN model, we tried 4 variations of the model, eliminating different elements one at a time. Figure 4 presents the results of the ablation tests. Removing either the label sequence optimization layer, pre-trained token embeddings, or token embeddings slightly decreased the performance. Surprisingly, the ANN performed pretty well with only character embeddings and without the token embeddings, and eliminating the character embeddings was more detrimental than eliminating the token embeddings. This suggests that the character-based token embeddings may be capturing not only the sub-token level features, but also the semantics of the tokens themselves.

Conclusions

We proposed the first system based on ANN for patient note de-identification. It outperforms state-of-the-art systems based on CRF on two datasets, while requiring no handcrafted features. Utilizing both the token and character embeddings, the system can automatically learn effective features from data by fine-tuning the parameters. It jointly learns the parameters for the embeddings, the bidirectional LSTMs as well as the label sequence optimization, and can make use of token embeddings pre-trained on large unlabeled datasets. Quantitative and qualitative analysis of the ANN and CRF models indicates that the ANN model better incorporates context and is more flexible to variations inherent in human languages than the CRF model.

From the viewpoint of deploying an off-the-shelf de-identification system, our results in Table 3 demonstrate recall on the MIMIC discharge summaries over 99%, which is quite encouraging. Figure 2, however, shows that the F1-score on the NAME category, probably the most sensitive PHI type, falls just below 98% for the ANN model. We anticipate that adding gazetteer features based on the local institution’s patient and staff census should improve this result, which will be explored in future work.

Funding

The project was supported by Philips Research. The content is solely the responsibility of the authors and does not necessarily represent the official views of Philips Research.

Acknowledgments

We warmly thank Michele Filannino, Alistair Johnson, and Tom Pollard for their helpful suggestions and technical assistance.

References