MeSH Term Suggestion for Systematic Review Literature Search

Introduction & Related Work

A medical systematic review is a comprehensive review of literature for a highly focused research question. Systematic reviews are seen as the highest form of evidence and are used extensively in healthcare decision making and clinical medical practice. In order to synthesise literature into a systematic review, a search must be undertaken. A major component of this search is a Boolean query. The Boolean query is often developed by a trained expert (i.e., an information specialist), who works closely with the research team to develop the search, and usually has knowledge about the domain.

The most commonly used database for searching medical literature is PubMed. Due to the increasing size and scope of the PubMed database, the Medical Subject Headings (MeSH) ontology was developed to conceptually index studies (Zieman and Bleich, 1997; Richter and Austin, 2012). MeSH is a controlled vocabulary thesaurus arranged in a hierarchical tree structure (specificity increases with depth in a parent$\rightarrow$child relationship, e.g., Anatomy$\rightarrow$Body Regions$\rightarrow$Head$\rightarrow$Eye…etc.). Indexing and categorising studies with MeSH terms enables queries to be developed which incorporate both free-text keywords and MeSH terms — enabling more effective searches. The use of MeSH terms in queries has been shown to be more effective than free-text keywords alone (Chang et al., 2006; Abdou and Savoy, 2008; Tenopir, 1985; Richter and Austin, 2012), e.g, they increase precision (Liu and Wacholder, 2017) and are far less ambiguous than free-text (Wacholder et al., 1997). However, it is still difficult even for expert information specialists to be familiar with the entire MeSH controlled vocabulary (Liu, 2009; Liu and Wacholder, 2017) — at the time of writing, MeSH contains 29,640 unique headings.

One way that PubMed has attempted to overcome this difficulty is by developing a method called Automatic Term Mapping (ATM). ATM is an automatic query expansion method which attempts to seamlessly map free-text keywords in a query to one of the three categories (index tables): MeSH, journal name or author name (Nahin, 2003). Although ATM is applied by default for all queries issued to PubMed, it has several semantic limitations: it is inaccurate when used to expand free-text acronyms into MeSH terms (Schulz et al., 2001); will produce different MeSH expansions even though synonymic free-text terms are used (Adlassnig et al., 2009), and has difficulty disambiguating between MeSH terms and journal names (Smith, 2004). Despite these limitations, the use of ATM for MeSH term suggestion has been shown to increase the precision of free-text searches in the genomic domain (Lu et al., 2009). However, its use has, to the best of the authors knowledge, not been empirically evaluated in the context of improving the effectiveness of systematic review literature search queries.

Our paper introduces the task of MeSH term suggestion for Boolean queries used in systematic review literature search. We model this task within the context of an information specialist looking for MeSH terms to add to a query without MeSH terms present. In addition to new MeSH suggestion methods, we also propose a framework to evaluate the effectiveness of the suggestion of MeSH terms on an established collection of systematic review literature search queries. This paper adds to a recent stream of research that has focused on computational methods for the assisted creation (Scells et al., 2020b, a, 2021) or refinement (Harrisen and Guido, 2018; Scells et al., 2019) of Boolean queries for systematic review creation.

The introduction of the new task of suggesting MeSH terms for systematic review literature search (Boolean queries), modeled within the context of an information specialist looking for MeSH terms to add to a query without MeSH terms present.

An empirical evaluation of the effectiveness of MeSH suggestion methods for this task (i.e., ranking MeSH terms for a query).

An empirical evaluation of the effectiveness of Boolean queries using the suggestions made by different suggestion methods (i.e., retrieving abstracts for a query given different suggested MeSH terms).

MeSH Term Suggestion

Next, we outline how we perform MeSH term suggestions for Boolean queries. As the Boolean queries used for systematic review literature search are highly complex, containing nested Boolean clauses, MeSH terms are suggested not globally, but instead locally, for query fragments. A query fragment is a clause of a Boolean query containing semantically related text clauses (i.e., free-text or MeSH terms). Each text clause in a query fragment is grouped into a Boolean clause using the OR operator. To give an intuition for how query fragments are derived and utilised for MeSH term suggestions, see Figure 1. The OR operators in Figure 1 are implicit. We exploit these fragments to perform a fine-grain evaluation for MeSH term suggestion (i.e., in terms of retrieval performance, ranking performance, and refinement of the ranking performance). However, we also perform defragmentation to obtain a Boolean query with suggested MeSH terms for comparison to the original Boolean queries.

We propose to suggest MeSH terms in a pipeline of three steps: retrieval, ranking, and refinement. The following three sections provide a description of how we approach each of these steps.

The first step in our MeSH term suggestion pipeline is the retrieval of MeSH terms. The retrieval of MeSH terms is facilitated by three different methods:

The entire free-text only query fragment is submitted to the PubMed entrez API (Sayers, 2010) for ATM. When free-text clauses without specific qualifiersQualifiers are the terminology PubMed uses for field restrictions. Keywords in a query may be explicitly restricted to certain fields, e.g., title, abstract, MeSH terms etc. are present in a query, the three index tables (MeSH, journal name, author name) are searched sequentially to determine if a mapping exists. If there is no mapping found initially, the free-text clause is divided into individual terms and the process is repeated. Mapped terms are filtered to only include those that are MeSH terms.

Each free-text clause in a query fragment is submitted to MetaMap (Aronson, 2001).Version 2018 with options set to default values. The results from MetaMap are filtered to only include those entities derived from the MeSH source. All of the mapped MeSH terms are recorded for each of the free-text terms in a query fragment. Additionally, the MetaMap score is recorded for each MeSH term.

We index the UMLS (Bodenreider, 2004) (version 2019AB) MRCONSO, MRDEF, MRREL, and MRSTY tables into Elasticsearch v7.6. Each free-text clause in the query fragment with MeSH terms removed is submitted to the Elasticsearch index. The results from the search are filtered to only include synonyms of concepts derived from the MeSH source. The synonyms of a concept are recorded for each term in the query fragment. Additionally, the BM25 score is recorded for each MeSH term (i.e., the default scoring mechanism of Elasticsearch).

For the MetaMap and UMLS approaches, the same MeSH term may be retrieved multiple times for a given free-text clause. To overcome this issue, we re-score the MeSH terms using rank fusion (CombSUM) (Fox and Shaw, 1994). The intuition for this re-scoring is that highly common MeSH terms that also obtain a high score from these retrieval methods should be scored highly overall (thus ranked higher than common MeSH terms and highly scoring MeSH terms).

2. MeSH Term Ranking

Once MeSH terms have been retrieved, they are ranked according to the approach for entity ranking described by Jimmy et al. (Jimmy et al., 2019) by adapting features proposed by Balog (Balog, 2018). In total, we use eleven features, each described in Table 1. For the description of MeSH terms ($d_{e}$), we scrape the corresponding Wikipedia page. We generate features for each MeSH term retrieval method (i.e., ATM, MetaMap, UMLS). Positive instances correspond to MeSH terms in the original query fragment, negative instances correspond to MeSH terms not in the original query fragment (binary labels). With features and instance labels, we train a learning-to-rank (LTR) model for each MeSH term retrieval method.

In addition to the LTR models, we also investigate a rank fusion approach (Fox and Shaw, 1994), where we combine the normalized MeSH term suggestion scores from each of the three methods to produce a new ranking that incorporates the highest ranking MeSH terms from each method. The intuition for investigating rank fusion in this context is that each method may retrieve different MeSH terms; and those terms may be ranked differently each time. Therefore, we wish to boost MeSH terms that are retrieved and ranked highly by each method, and further boost those MeSH terms retrieved by multiple methods.

3. MeSH Term Refinement

Finally, we seek to refine the suggested MeSH terms by estimating a rank cut-off. By refinement, we mean to limit the number of MeSH terms to only the most applicable for a query fragment. We do this using a score-based gain function which models gain as the score for a MeSH term. Formally, the cumulative gain $CG$ for a MeSH term at rank $p$ is $CG_{p}=\sum_{i=1}^{p}score_{i}$; where the score for a MeSH term is equal to $1-normalised\ score$ (i.e., min-max normalisation) for the MeSH term.

We tune a $\kappa$ parameter for each retrieval method which controls the percentage of total $CG$ allowed to be observed before the ranking is cut-off (i.e., a refinement of the ranking). The $\kappa$ parameter is tuned from 5% to 95% in increments of 5%. The intuition for re-scoring MeSH terms becomes apparent when used with the $\kappa$ parameter: the highest-ranking MeSH term will receive a score of 0, resulting in at least one MeSH term suggested for every query fragment.

Note that MeSH terms may share the same score, i.e., they may be tied. We take a conservative approach to account for the problem of tied MeSH terms at the boundary of the cut-off specified by $\kappa$. Whenever we encounter ties, we treat all of the tied MeSH terms as a single accumulation of gain that equals the summed gain across the scores of the tied MeSH terms. This treatment has the effect that tied MeSH terms account for much larger accumulations of gain. Therefore, tied MeSH terms at the top of rankings are more likely to be included in the cut-off than tied MeSH terms at the bottom. In essence, either all tied MeSH terms are considered within the cut-off (i.e., ties at the top of the ranking), or no tied MeSH terms are considered (i.e., ties at the bottom of the ranking).

4. Evaluation

We evaluate the effectiveness of MeSH term suggestions retrospectively using the MeSH terms identified from pre-existing queries as a gold standard. In doing so, we make the assumption that the MeSH terms in these pre-existing queries are the ideal choices. As such, this gold standard may be biased to favour the PubMed ATM method, as it could have been used to suggest MeSH terms originally. The MeSH term suggestion methods proposed above are likely to identify MeSH terms that were not originally in pre-existing query fragments. To combat this assumption, we also evaluate the retrieval effectiveness achieved by the queries with the proposed suggestions. We therefore evaluate both (i) the effectiveness of query suggestion given the assumption that MeSH terms in pre-existing queries are a gold standard; and (ii) the effectiveness of the query at retrieving studies.

Note that (ii) also has limitations: that query fragments must be combined back into the original query structure in order to properly evaluate the query; and new MeSH terms may retrieve studies that are unjudged (it is unknown if the retrieved unjudged studies are relevant or not). To account for these unjudged studies, we use the approach proposed by Scells et al. (Scells et al., 2019), which calculates, in addition to the lower bound typically assumed (i.e., all unjudged studies are irrelevant), an upper bound (i.e., assume all unjudged studies are relevant) and a balance between the two (i.e., assume some unjudged studies to be relevant given a maximum likelihood estimation over the judged studies). Note that in the paper, mle method randomly sampled unjudged studies be relevant using maximum likelihood ratio, which is equivalent to the ratio of relevant studies in the original candidate documents.

The effectiveness of the MeSH term suggestion is evaluated using reciprocal rank, nDCG@{5,10}, recall@{5,10}, precision, and recall. Precision and recall measure the effectiveness of the retrieval of MeSH terms by the three retrieval methods. nDCG and reciprocal rank measure the effectiveness of the LTR entity ranking model for each of the three retrieval models.

To evaluate the effectiveness of the suggested MeSH terms for the task of systematic review literature search, once query fragments are defragmented, the retrieval effectiveness is evaluated using typical systematic review literature search measures: precision, recall, and Fβ={0.5,1,3}, note we will not report results for Fβ={0.5,3} as their generally trend stay the same with Fβ={1}. To obtain retrieval results, the PubMed entrez API is used to directly issue defragmented Boolean queries. For reproducibility purposes, as PubMed is constantly updated with new studies, we apply a date restriction to all queries.

For both evaluation settings (i.e., ranking MeSH term suggestion and Boolean query retrieval), we evaluate the quality of ranking in two settings: (i) all, where all retrieved MeSH terms are considered; and (i) cut, where a score-based cut-off is determined to filter the suggested MeSH terms.

Experimental Setup

We use topics from the CLEF TAR task from 2017, 2018, and 2019 (Kanoulas et al., 2017, 2018, 2019). 15 topics are discarded due to lack of MeSH terms (2017: CD007427, CD010771, CD010772, CD010775, CD010783, CD010860, CD011145; 2018: CD007427, CD009263, CD009694; 2019: CD006715, CD007427, CD009263, CD009694, CD011768). An additional 5 topics are discarded because of retrieval issues (2017: CD010276, CD010173, CD012019; 2018: CD011926; 2019: CD010038), likely resulting from the fact that some queries are automatically translated from queries in one format (Ovid Medline) into another format (PubMed). In total we used 242 topics across all three datasets (114 unique, as each year has partial overlap). For each topic, we divide the Boolean query for that topic into several query fragments. We create these fragments using the transmute tool (Scells et al., 2018). Each fragment contains at least one MeSH term. This results in a total of 302 unique query fragments for the three years (2.65 fragments per query on average). For each of the query fragments, we corrected any errors (e.g., spelling mistakes, syntactic errors), extracted MeSH terms, keywords, query fragment with MeSH terms, and query fragments without MeSH terms. For training the LTR model for MeSH term ranking, the pre-split training and test portions from the CLEF datasets are used. The 2019 topics are split also on systematic review type (intervention and diagnostic test accuracy — indicated as I and D respectively in the results), while those for 2017 and 2018 are all diagnostic test accuracy. We use the quickrank library (Capannini et al., 2016) for LTR, instantiated with LambdaMART trained to maximise nDCG. We leave other settings as per default.

Results

All of the results in this section are presented on the testing portions of each CLEF TAR year (i.e., 2017, 2018, 2019/I, 2019/D).

Firstly, we investigate the MeSH term suggestion methods’ effectiveness in retrieving terms given a query fragment. Table 2 reports precision (P) and recall (R) for the retrieval of MeSH terms. When comparing the three base retrieval methods, UMLS generally retrieves more relevant terms than ATM and MetaMap, as suggested by the higher recall value for UMLS than the other two methods. However, the UMLS method achieves lower precision than the other two methods, indicating that it retrieves too many MeSH terms. The fusion method achieves the highest recall across all datasets. However, it never outperforms the other methods in terms of precision (naturally because it combines all the MeSH suggestions). We find that: (i) UMLS is the most effective MeSH retrieval method for recall, (ii) ATM is the most effective retrieval method for precision, and (iii) that fusion of multiple MeSH retrieval methods generally leads to the highest recall and lowest precision.

1.2. Ranking of MeSH Terms

Next, we investigate the effectiveness of the LTR model at ranking the retrieved MeSH terms for each retrieval method. For this task, we observe the reciprocal rank (RR), R@k (Recall@k), and nDCG@k of the results reported in Table 2. We find that (i) because the UMLS method generally has the highest recall compared to ATM and MetaMap, the ranking performance was also generally higher than these methods in most measures and (ii) also, due to the higher recall, the fusion method produces more effective rankings of MeSH terms, except for RR on the 2018 dataset.

1.3. Refinement of MeSH Terms

Finally, we investigate the effect of refining the ranked MeSH terms by cutting off the ranking at a certain point and discarding the remainder. We estimate this cut-off point through a parameter. Our tuning results on the training portions of the datasets are presented in Figure 2. We believe that the spikes in these plots generally correspond to the inclusion and exclusion of ties. These spikes are most prominent in the MetaMap, and ATM methods as these methods do not assign highly discriminative scores to MeSH terms. Furthermore, note that the UMLS and fusion methods have considerably smoother shapes, as these methods have highly discriminative scores.

We investigate the effect that this refinement has on the MeSH term suggestion performance in Table 2 (i.e., with -C). We find that refinement generally improves precision while lowering recall. The loss in recall attributed to the refinement negatively affects ranking effectiveness. In most cases, refinement is worse than ranking all of the MeSH terms and often significantly worse than the ATM baseline (without refinement).

2. Search Effectiveness

We next investigate the impact in performance that MeSH term retrieval, ranking, and ranking refinement has on the retrieval effectiveness of Boolean queries.

Comparing the original query to the original query with MeSH terms removed, a general trend in every dataset is that the removal of MeSH terms results in a tradeoff where precision increases and recall decreases. Comparing the original query with the MeSH term suggestion methods, the same trend also appears, which indicates that the addition of relevant MeSH terms will increase the number of relevant studies.

The retrieval of literature for systematic reviews is a high recall task. Our MeSH term suggestion methods can automatically support this high recall task by recommending appropriate MeSH terms given a Boolean query without MeSH terms. The results obtained from many of the MeSH term suggestion methods presented in this work are comparable to those achieved by queries initially constructed by information specialists.

2.2. Impact of Unjudged Studies

We next examine the retrieval effectiveness when we consider unjudged studies to be irrelevant. This assumption is a typical retrieval evaluation scenario and provides a lower bound on effectiveness. For the 2017 and 2018 datasets, we find that few methods increase precision over the original queries (both are refined rankings); however, for the two 2019 datasets, there is no method where we see an increase in precision over the original queries. However, none of the results obtained statistically significantly worse results except for 2017/F.

Comparing these results to our optimistic and MLE residual treatments of unjudged studies, we find that (i) the unrefined fusion ranking achieves the highest results in recall across all datasets, likely a result of the fact that it retrieves the most MeSH terms; (ii) Although we find that unrefined fusion still achieves the highest recall for the MLE treatment, it generally performs worse than other methods.

2.3. Impact of Fusion

In terms of recall, the unrefined fusion ranking improved recall except for a single case (2018). This large gain in recall is likely because unrefined fusion combines all of the MeSH terms suggested by the other three methods (ATM, MetaMap, and UMLS). This suggests that the unrefined fusion method is not beneficial for improving the precision of a Boolean query. However, suppose semi-automatic MeSH term suggestion can be used. Information specialists may be able to use the suggestion and apply their expertise to decide which MeSH terms should be included to achieve higher performance.

To our surprise, the refined fusion method did not achieve the highest result among any evaluation measure or dataset. Indeed, refinement of rankings generally lowered recall and had a negligible effect on precision for all methods. This suggests that choosing appropriate MeSH terms is crucial for effective systematic review literature retrieval instead of adding as many MeSH terms as possible.

3. Case Study

Based on the findings in Section 4.2, it may also be interesting to observe a concrete example of MeSH term suggestion and why the effectiveness may vary for different suggestion methods. After exploring different queries produced from our MeSH suggestion methods, we chose CD010542 from the 2017 CLEF TAR dataset to conduct our analysis because it is a representative example. We follow the same procedure as in Figure 1 by removing all MeSH terms, fragmenting the query into separate clauses, suggesting MeSH terms for each clause, and defragmenting the clauses with suggested MeSH into a Boolean query we use for retrieval. We use the re-constructed queries to search PubMed and compare their effectiveness with the original query and the query without any MeSH terms. Query fragments in the procedure and suggested method are shown in Table 4.

In Table 5, we found that the scores obtained using ATM and Metamap are the same while their first fragment is different. This strengthens our hypothesis from Section 4.2.3 that the choice of MeSH terms is crucial to the effectiveness of a Boolean query in systematic review literature search. When comparing UMLS with other methods, we found that UMLS obtained a higher precision when compare with other query with MeSH terms except when optimistic measurements were used, this suggests that the MeSH term from fragment 1 in the original query may be detrimental to the effectiveness of the query; therefore, even the MeSH terms in the original queries may not be the most effective choice.

Finally, the query with MeSH terms removed achieved a higher precision than any other methods and maintained the same recall value, this suggests that MeSH terms may not be effective to some queries, a more dynamic method to add the most appropriate MeSH terms may have the potential to further improve performance of MeSH term suggestion. We propose that: (i) For a fully automatic pipeline, a classification model can be trained and used to decide the effectiveness gain of a MeSH term, a stopping strategy can also be used to make the best decision of when to stop adding terms to obtain the best performance. (ii) For a semi-automatic pipeline, the classification model can be reused to compute MeSH terms’ confidence scores recurrently, this may provide a better understanding for information specialists to decide on which MeSH terms to use when constructing the new query.

Conclusions

In this paper, we presented the new task of suggesting MeSH terms within the context of systematic review literature search (suggestion for Boolean queries). We provided a comprehensive evaluation of the effectiveness of MeSH suggestion methods (in terms of retrieval, ranking, and refinement). We compared these methods to the existing method that PubMed uses to suggest MeSH terms (ATM). We found that both the MetaMap and UMLS suggestion methods can improve the retrieval effectiveness of Boolean queries. Unsurprisingly, when we combined the three methods using rank fusion, we found the highest gains in retrieval effectiveness.

Our methods overcome the semantic limitations of ATM: the MetaMap and UMLS methods both suggested more relevant MeSH terms than ATM, and the addition of these terms positively impacted retrieval performance. Often this came with a minor loss in recall. Note that there are generally between 10-100 relevant studies per topic: the actual impact of loss in recall is attributed to only a handful of studies and is likely not to impact the results of a systematic review.

Identifying MeSH terms to add to a Boolean query for systematic review literature search is known to be a difficult task for humans to accomplish. The outcomes of this paper have implications for both the information retrieval and systematic review communities. Firstly, our methods can be used in automatic query formulation situations (see, e.g., tasks in CLEF TAR). Secondly, they can be integrated into existing tools to assist information specialists in formulating more effective queries (Scells and Zuccon, 2018; Li et al., 2020).

This research is supported by the Australian Research Council (DP210104043). Dr Guido Zuccon is the recipient of an Australian Research Council DECRA Research Fellowship (DE180101579).

References