Evolution of Semantic Similarity -- A Survey

Introduction

With the exponential increase in text data generated over time, Natural Language Processing (NLP) has gained significant attention from Artificial Intelligence (AI) experts. Measuring the semantic similarity between various text components like words, sentences, or documents plays a significant role in a wide range of NLP tasks like information retrieval (Kim et al., 2017), text summarization (Mohamed and Oussalah, 2019), text classification (Kim, 2014), essay evaluation (Janda et al., 2019), machine translation (Zou et al., 2013), question answering (Bordes et al., 2014; Lopez-Gazpio et al., 2017), among others. In the early days, two text snippets were considered similar if they contain the same words/characters. The techniques like Bag of Words (BoW), Term Frequency - Inverse Document Frequency (TF-IDF) were used to represent text, as real value vectors to aid calculation of semantic similarity. However, these techniques did not attribute to the fact that words have different meanings and different words can be used to represent a similar concept. For example, consider two sentences “John and David studied Maths and Science.” and “John studied Maths and David studied Science.” Though these two sentences have exactly the same words they do not convey the same meaning. Similarly, the sentences “Mary is allergic to dairy products.” and “Mary is lactose intolerant.” convey the same meaning; however, they do not have the same set of words. These methods captured the lexical feature of the text and were simple to implement, however, they ignored the semantic and syntactic properties of text. To address these drawbacks of the lexical measures various semantic similarity techniques were proposed over the past three decades. Semantic Textual Similarity (STS) is defined as the measure of semantic equivalence between two blocks of text. Semantic similarity methods usually give a ranking or percentage of similarity between texts, rather than a binary decision as similar or not similar. Semantic similarity is often used synonymously with semantic relatedness. However, semantic relatedness not only accounts for the semantic similarity between texts but also considers a broader perspective analyzing the shared semantic properties of two words. For example, the words ‘coffee’ and ‘mug’ may be related to one another closely, but they are not considered semantically similar whereas the words ‘coffee’ and ‘tea’ are semantically similar. Thus, semantic similarity may be considered, as one of the aspects of semantic relatedness. The semantic relationship including similarity is measured in terms of semantic distance, which is inversely proportional to the relationship (Hadj Taieb et al., 2019).

Most of the survey articles published recently related to semantic similarity, provide in-depth knowledge of one particular semantic similarity technique or a single application of semantic similarity. Lastra-Díaz et al. survey various knowledge-based methods (Lastra-Díaz et al., 2019) and IC-based methods (Lastra-Díaz and García-Serrano, 2015), Camacho-Colladas et al. (Camacho-Collados and Pilehvar, 2018) discuss various vector representation methods of words, Taieb et al. (Hadj Taieb et al., 2019), on the other hand, describe various semantic relatedness methods and Berna Altınel et al. (Altınel and Ganiz, 2018) summarise various semantic similarity methods used for text classification. The motivation behind this survey is to provide a comprehensive account of the various semantic similarity techniques including the most recent advancements using deep neural network-based methods. This survey traces the evolution of Semantic Similarity Techniques over the past decades, distinguishing them based on the underlying methods used in them. Figure 1 shows the structure of the survey. A detailed account of the widely used datasets available for semantic similarity is provided in Section 2. Sections 3 to 6 provide a detailed description of semantic similarity methods broadly classified as 1) Knowledge-based methods, 2) Corpus-based methods, 3) Deep neural network-based methods, and 4) Hybrid methods. Section 7 analyzes the various aspects and inference of the survey conducted. This survey provides a deep and wide knowledge of existing techniques for new researchers who venture to explore one of the most challenging NLP tasks, Semantic Textual Similarity.

Datasets

In this section, we discuss some of the popular datasets used to evaluate the performance of semantic similarity algorithms. The datasets may include word pairs or sentence pairs with associated standard similarity values. The performance of various semantic similarity algorithms is measured by the correlation of the achieved results with that of the standard measures available in these datasets. Table LABEL:tab:Dataset_list lists some of the popular datasets used to evaluate the performance of semantic similarity algorithms. The below subsection describes the attributes of the dataset and the methodology used to construct them.

The following is a list of widely used semantic similarity datasets arranged chronologically.

Rubenstein and Goodenough (R&G) (Rubenstein and Goodenough, 1965): This dataset was created as a result of an experiment conducted among 51 undergraduate students (native English speakers) in two different sessions. The subjects were provided with 65 selected English noun pairs and requested to assign a similarity score for each pair over a scale of 0 to 4, where 0 represents that the words are completely dissimilar and 4 represents that they are highly similar. This dataset is the first and most widely used dataset in semantic similarity tasks (Zhu and Iglesias, 2017).

Miller and Charles (M&C) (Miller and Charles, 1991): Miller and Charles repeated the experiment performed by Rubenstein and Goodenough in 1991 with a subset of 30 word pairs from the original 65 word pairs. 38 human subjects ranked the word pairs on a scale from 0 to 4, 4 being the ”most similar.”

WS353 (Finkelstein et al., 2001): WS353 contains 353 word pairs with an associated score ranging from 0 to 10. 0 represents the least similarity and 10 represents the highest similarity. The experiment was conducted with a group of 16 human subjects. This dataset measures semantic relatedness rather than semantic similarity. Subsequently, the next dataset was proposed.

WS353-Sim (Agirre et al., 2009): This dataset is a subset of WS353 containing 203 word pairs from the original 353 word pairs that are more suitable for semantic similarity algorithms specifically.

LiSent (Li et al., 2006): 65 sentence pairs were built using the dictionary definition of 65 word pairs used in the R&G dataset. 32 native English speakers volunteered to provide a similarity range from 0 to 4, 4 being the highest. The mean of the scores given by all the volunteers was taken as the final score.

SRS (Pedersen et al., 2007): Pedersen et al. (Pedersen et al., 2007) attempted to build a domain specific semantic similarity dataset for the biomedical domain. Initially 120 pairs were selected by a physician distributed with 30 pairs over 4 similarity values. These term pairs were then ranked by 13 medical coders on a scale of 1-10. 30 word pairs from the 120 pairs were selected to increase reliability and these word pairs were annotated by 3 physicians and 9 (out of the 13) medical coders to form the final dataset.

SimLex-999 (Hill et al., 2015): 999 word pairs were selected from the UFS Dataset (Nelson et al., 2004) of which 900 were similar and 99 were related but not similar. 500 native English speakers, recruited via Amazon Mechanical Turk were asked to rank the similarity between the word pairs over a scale of 0 to 6, 6 being the most similar. The dataset contains 666 noun pairs, 222 verb pairs, and 111 adjective pairs.

Sentences Involving Compositional Knowledge (SICK) dataset (Marelli et al., 2014): The SICK dataset consists of 10,000 sentence pairs, derived from two existing datasets the ImageFlickr 8 and MSR-Video descriptions dataset. Each sentence pair is associated with a relatedness score and a text entailment relation. The relatedness score ranges from 1 to 5, and the three entailment relations are ”NEUTRAL, ENTAILMENT and CONTRADICTION.” The annotation was done using crowd-sourcing techniques.

STS datasets (Agirre et al., 2012; Agirre et al., 2013; Agirre et al., 2014; Agirre et al., 2015, 2016; Cer et al., 2017): The STS datasets were built by combining sentence pairs from different sources by the organizers of the SemEVAL shared task. The dataset was annotated using Amazon Mechanical Turk and further verified by the organizers themselves. Table LABEL:tab:ststable shows the various sources from which the STS dataset was built.

Knowledge-based Semantic-Similarity Methods

Knowledge-based semantic similarity methods calculate semantic similarity between two terms based on the information derived from one or more underlying knowledge sources like ontologies/lexical databases, thesauri, dictionaries, etc. The underlying knowledge-base offers these methods a structured representation of terms or concepts connected by semantic relations, further offering an ambiguity free semantic measure, as the actual meaning of the terms, is taken into consideration (Sánchez et al., 2012). In this section, we discuss four lexical databases widely employed in knowledge-based semantic similarity methods and further discuss in brief, different methodologies adopted by some of the knowledge-based semantic similarity methods.

WordNet (Miller, 1995) is a widely used lexical database for knowledge-based semantic similarity methods that accounts for more than 100,000 English concepts (Sánchez et al., 2012). WordNet can be visualized as a graph, where the nodes represent the meaning of the words (concepts), and the edges define the relationship between the words (Zhu and Iglesias, 2017). WordNet’s structure is primarily based on synonyms, where each word has different synsets attributed to their different meanings. The similarity between two words depends on the path distance between them (Pawar and Mago, 2019).

Wiktionaryhttps://en.wiktionary.org is an open-source lexical database that encompasses approximately 6.2 million words from 4,000 different languages. Each entry has an article page associated with it, and it accounts for a different sense of each entry. Wiktionary does not have a well-established taxonomic lexical relationship within the entries, unlike WordNet, which makes it difficult to be used in semantic similarity algorithms (Pilehvar and Navigli, 2015).

With the advent of Wikipediahttp://www.wikipedia.org, most techniques for semantic similarity exploit the abundant text data freely available to train the models (Mihalcea and Csomai, 2007). Wikipedia has the text data organized as Articles. Each article has a title (concept), neighbors, description, and categories. It is used as both structured taxonomic data and/or as a corpus for training corpus-based methods (Qu et al., 2018). The complex category structure of Wikipedia is used as a graph to determine the Information Content of concepts, which in turn aids in calculating the semantic similarity (Jiang et al., 2017).

BabelNet (Navigli and Ponzetto, 2012) is a lexical resource that combines WordNet with data available on Wikipedia for each synset. It is the largest multilingual semantic ontology available with nearly over 13 million synsets and 380 million semantic relations in 271 languages. It includes over four million synsets with at least one associated Wikipedia page for the English language (Camacho-Collados et al., 2016).

2. Types of Knowledge-based semantic similarity methods

Based on the underlying principle of how the semantic similarity between words is assessed, knowledge-based semantic similarity methods can be further categorized as edge-counting methods, feature-based methods, and information content-based methods.

The most straight forward edge counting method is to consider the underlying ontology as a graph connecting words taxonomically and count the edges between two terms to measure the similarity between them. The greater the distance between the terms the less similar they are. This measure called $path$ was proposed by Rada et al. (Rada et al., 1989) where the similarity is inversely proportional to the shortest path length between two terms. In this edge-counting method, the fact that the words deeper down the hierarchy have a more specific meaning, and that, they may be more similar to each other even though they have the same distance as two words that represent a more generic concept was not taken into consideration. Wu and Palmer (Wu and Palmer, 1994) proposed $wup$ measure, where the depth of the words in the ontology was considered an important attribute. The $wup$ measure counts the number of edges between each term and their Least Common Subsumer (LCS). LCS is the common ancestor shared by both terms in the given ontology. Consider, two terms denoted as $t_{1},t_{2}$, their LCS denoted as $t_{lcs}$, and the shortest path length between them denoted as $min\_len(t_{1},t_{2})$, $path$ is measured as,

Li et al. (Li et al., 2003) proposed a measure that takes into account both the minimum path distance and depth. $li$ is measured as,

However, the edge-counting methods ignore the fact that the edges in the ontologies need not be of equal length. To overcome this shortcoming of simple edge-counting methods feature-based semantic similarity methods were proposed.

2.2. Feature-based methods:

The feature-based methods calculate similarity as a function of properties of the words, like gloss, neighboring concepts, etc. (Sánchez et al., 2012). Gloss is defined as the meaning of a word in a dictionary; a collection of glosses is called a glossary. There are various semantic similarity methods proposed based on the gloss of words. Gloss-based semantic similarity measures exploit the knowledge that words with similar meanings have more common words in their gloss. The semantic similarity is measured as the extent of overlap between the gloss of the words in consideration. The Lesk measure (Banerjee and Pedersen, 2003), assigns a value of relatedness between two words based on the overlap of words in their gloss and the glosses of the concepts they are related to in an ontology like WordNet (Lastra-Díaz et al., 2019). Jiang et al. (Jiang et al., 2015) proposed a feature-based method where semantic similarity is measured using the glosses of concepts present in Wikipedia. Most feature-based methods take into account common and non-common features between two words/terms. The common features contribute to the increase of the similarity value and the non-common features decrease the similarity value. The major limitation of feature-based methods is its dependency on ontologies with semantic features, and most ontologies rarely incorporate any semantic features other than taxonomic relationships (Sánchez et al., 2012).

2.3. Information Content-based methods:

Information content (IC) of a concept is defined as the information derived from the concept when it appears in context (Sánchez and Batet, 2013). A high IC value indicates that the word is more specific and clearly describes a concept with less ambiguity, while lower IC values indicate that the words are more abstract in meaning (Zhu and Iglesias, 2017). The specificity of the word is determined using Inverse Document Frequency (IDF), which relies on the principle that the more specific a word is, the less it occurs in a document. Information content-based methods measure the similarity between terms using the IC value associated with them. Resnik and Philip (Resnik, 1995) proposed a semantic similarity measure called $res$ which measures the similarity based on the idea that if two concepts share a common subsumer they share more information since the $IC$ value of the LCS is higher. Considering $IC$ represents the Information Content of the given term, $res$ is measured as,

D. Lin (Lin et al., 1998) proposed an extension of the $res$ measure taking into consideration the $IC$ value of both the terms that attribute to the individual information or description of the terms and the $IC$ value of their LCS that provides the shared commonality between the terms. $lin$ is measured as,

Jiang and Conrath (Jiang and Conrath, 1997) calculate a distance measure based on the difference between the sum of the individual $IC$ values of the terms and the $IC$ value of their LCS using the below equation,

The distance measure replaces the shortest path length in equation (1), and the similarity is inversely proportional to the above distance. Hence $jcn$ is measured as,

IC can be measured using an underlying corpora or from the intrinsic structure of the ontology itself (Sánchez et al., 2011) based on the assumption that the ontologies are structured in a meaningful way. Some of the terms may not be included in one ontology, which provides a scope to use multiple ontologies to calculate their relationship (Rodríguez and Egenhofer, 2003). Based on whether the given terms are both present in a single ontology or not, IC-based methods can be classified as mono-ontological methods or multi-ontological methods. When multiple ontologies are involved the $IC$ of the Least Common Subsumer from both the ontologies are accessed to estimate the semantic similarity values. Jiang et al. (Jiang et al., 2017) proposed IC-based semantic similarity measures based on Wikipedia pages, concepts and neighbors. Wikipedia was both used as a structured taxonomy as well as a corpus to provide $IC$ values.

2.4. Combined knowledge-based methods:

Various similarity measures were proposed combining the various knowledge-based methods. Goa et al. (Gao et al., 2015) proposed a semantic similarity method based on WordNet ontology where three different strategies are used to add weights to the edges and the shortest weighted path is used to measure the semantic similarity. According to the first strategy, the depths of all the terms in WordNet along the path between the two terms in consideration is added as a weight to the shortest path. In the second strategy, only the depth of the LCS of the terms was added as the weight, and in strategy three, the $IC$ value of the terms is added as weight. The shortest weighted path length is now calculated and then non-linearly transformed to produce semantic similarity measures. In comparison, it is shown that strategy three achieved a better correlation to the gold standards in comparison with traditional methods and the two other strategies proposed. Zhu and Iglesias (Zhu and Iglesias, 2017) proposed another weighted path measure called $wpath$ that adds the $IC$ value of the Least Common Subsumer as a weight to the shortest path length. $wpath$ is calculated as

This method was proposed to be used in various knowledge graphs (KG) like WordNet (Miller, 1995), DBPedia (Bizer et al., 2009), YAGO (Hoffart et al., 2013), etc. and the parameter $k$ is a hyperparameter which has to be tuned for different KGs and different domains as different KGs have a different distribution of terms in each domain. Both corpus-based IC and intrinsic IC values were experimented and corpus IC-based $wpath$ measure achieved greater correlation in most of the gold standard datasets.

Knowledge-based semantic similarity methods are computationally simple, and the underlying knowledge-base acts as a strong backbone for the models, and the most common problem of ambiguity like synonyms, idioms, and phrases are handled efficiently. Knowledge-based methods can easily be extended to calculate sentence to sentence similarity measure by defining rules for aggregation (Lee, 2011). Lastra-Díaz et al. (Lastra-Díaz et al., 2017) developed a software Half-Edge Semantic Measures Library (HESML) to implement various ontology-based semantic similarity measures proposed and have shown an increase in performance time and scalability of the models.

However, knowledge-based systems are highly dependent on the underlying source resulting in the need to update them frequently which requires time and high computational resources. Although strong ontologies like WordNet, exist for the English language, similar resources are not available for other languages that results in the need for the building of strong and structured knowledge bases to implement knowledge-based methods in different languages and across different domains. Various research works were conducted on extending semantic similarity measures in the biomedical domain (Pedersen et al., 2007; Soğancıoğlu et al., 2017). McInnes et al. (McInnes et al., [n.d.]) built a domain-specific model called UMLS to measure the similarity between words in the biomedical domain. With nearly 6,500 world languages and numerous domains, this becomes a serious drawback for knowledge-based systems.

Corpus-based Semantic-Similarity Methods

Corpus-based semantic similarity methods measure semantic similarity between terms using the information retrieved from large corpora. The underlying principle called ‘distributional hypothesis’ (Gorman and Curran, 2006) exploits the idea that ”similar words occur together, frequently”; however, the actual meaning of the words is not taken into consideration. While various techniques were used to construct the vector representation of the text data, several semantic distance measures based on the distributional hypothesis were proposed to estimate the similarity between the vectors. A comprehensive survey of various distributional semantic measures was carried out by Mohammad and Hurst (Mohammad and Hirst, 2012), and the different measure and their respective formula are provided in Table LABEL:tab:tab4 in Appendix A . However, among all these measures, the cosine similarity gained significance and has been widely used among NLP researchers to date (Mohammad and Hirst, 2012). In this section, we discuss in detail some of the widely used word-embeddings built using distributional hypothesis and some of the significant corpus-based semantic similarity methods.

Word embeddings provide vector representations of words wherein these vectors retain the underlying linguistic relationship between the words (Schnabel et al., 2015). These vectors are computed using different approaches like neural networks (Mikolov et al., 2013a), word co-occurrence matrix (Pennington et al., 2014), or representations in terms of the context in which the word appears (Levy and Goldberg, 2014a). Some of the most widely used pre-trained word embeddings include:

word2vec (Mikolov et al., 2013a): Developed from Google News dataset, containing approximately 3 million vector representations of words and phrases, $word2vec$ is a neural network model used to produce distributed vector representation of words based on an underlying corpus. There are two different models of $word2vec$ proposed: the Continuous Bag of Words (CBOW) and the Skip-gram model. The architecture of the network is rather simple and contains an input layer, one hidden layer, and an output layer. The network is fed with a large text corpus as the input, and the output of the model is the vector representations of words. The CBOW model predicts the current word using the neighboring context words, while the Skip-gram model predicts the neighboring context words given a target word. $word2vec$ models are efficient in representing the words as vectors that retain the contextual similarity between words. The word vector calculations yielded good results in predicting the semantic similarity (Mikolov et al., 2013b). Many researchers extended the $word2vec$ model to propose context vectors (Melamud et al., 2016), dictionary vectors (Tissier et al., 2017), sentence vectors (Pagliardini et al., 2018) and paragraph vectors (Le and Mikolov, 2014).

GloVe (Pennington et al., 2014): $GloVe$ developed by Stanford University relies on a global word co-occurrence matrix formed based on the underlying corpus. It estimates similarity based on the principle that words similar to each other occur together. The co-occurrence matrix is populated with occurrence values by doing a single pass over the underlying large corpora. $GloVe$ model was trained using five different corpora mostly Wikipedia dumps. While forming vectors, words are chosen within a specified context window owing to the fact that words far away have less relevance to the context word in consideration. The $GloVe$ loss function minimizes the least-square distance between the context window co-occurrence values and the global co-occurrence values (Lastra-Díaz et al., 2019). $GloVe$ vectors were extended to form contextualized word vectors to differentiate words based on context (McCann et al., 2017).

fastText (Bojanowski et al., 2017): Facebook AI researchers developed a word embedding model that builds word vectors based on Skip-gram models where each word is represented as a collection of character n-grams. $fastText$ learns word embeddings as the average of its character embeddings thus accounting for the morphological structure of the word which proves efficient in various languages like Finnish and Turkish. Even out-of-the-vocabulary words are assigned word vectors based on their characters or subunits.

Bidirectional Encoder Representations from Transformers(BERT) (Devlin et al., 2019): Devlin et al. (Devlin et al., 2019) proposed a pretrained transformer-based word embeddings which can be fine-tuned by adding a final output layer to accommodate the embeddings to different NLP tasks. BERT uses the transformer architecture proposed by Vaswani et al. (Vaswani et al., 2017), which produces attention-based word vectors using a bi-directional transformer encoder. The BERT framework involves two important processes namely ‘pre-training’ and ‘fine-tuning’. The model is pretrained using a corpus of nearly 3,300M words from both the Book corpus and English Wikipedia. Since the model is bidirectional in order to avoid the possibility of the model knowing the token itself when training from both directions the pretraining process is carried out in two different ways. In the first task, random words in the corpus are masked and the model is trained to predict these words. In the second task, the model is presented with sentence pairs from the corpus, in which 50 percent of the sentences are actually consecutive while the remaining are random pairs. The model is trained to predict if the given sentence pair are consecutive or not. In the ‘fine-tuning’ process, the model is trained for the specific down-stream NLP task at hand. The model is structured to take as input both single sentences and multiple sentences to accommodate a variety of NLP tasks. To train the model to perform a question answering task, the model is provided with various question-answer pairs and all the parameters are fine-tuned in accordance with the task. BERT embeddings provided state-of-the-art results in the STS-B data set with a Spearman’s correlation of 86.5% outperforming other BiLSTM models including ELMo (Peters et al., 2018).

Word embeddings are used to measure semantic similarity between texts of different languages by mapping the word embedding of one language over the vector space of another. On training with a limited yet sufficient number of translation pairs, the translation matrix can be computed to enable the overlap of embeddings across languages (Glavaš et al., 2018). One of the major challenges faced when deploying word-embeddings to measure similarity is Meaning Conflation Deficiency. It denotes that word embeddings do not attribute to the different meanings of a word that pollutes the semantic space with noise by bringing irrelevant words closer to each other. For example, the words ‘finance’ and ‘river’ may appear in the same semantic space since the word ‘bank’ has two different meanings (Camacho-Collados and Pilehvar, 2018). It is critical to understand that word-embeddings exploit the distributional hypothesis for the construction of vectors and rely on large corpora, hence, they are classified under corpus-based semantic similarity methods. However, deep-neural network based-methods and most hybrid semantic similarity methods use word-embeddings to convert the text data to high dimensional vectors, and the efficiency of these embeddings plays a significant role in the performance of the semantic similarity methods (Mnih and Kavukcuoglu, 2013; Levy and Goldberg, 2014b).

2. Types of corpus-based semantic similarity methods

Based on the underlying methods using which the word-vectors are constructed there are a wide variety of corpus-based methods some of which are discussed in this section.

LSA is one of the most popular and widely used corpus-based techniques used for measuring semantic similarity. A word co-occurrence matrix is formed where the rows represent the words and columns represent the paragraphs, and the cells are populated with word counts. This matrix is formed with a large underlying corpus, and dimensionality reduction is achieved by a mathematical technique called Singular Value Decomposition (SVD). SVD represents a given matrix as a product of three matrices, where two matrices represent the rows and columns as vectors derived from their eigenvalues and the third matrix is a diagonal matrix that has values that would reproduce the original matrix when multiplied with the other two matrices (Landauer et al., 1998). SVD reduces the number of columns while retaining the number of rows thereby preserving the similarity structure among the words. Then each word is represented as a vector using the values in its corresponding rows and semantic similarity is calculated as the cosine value between these vectors. LSA models are generalized by replacing words with texts and columns with different samples and are used to calculate the similarity between sentences, paragraphs, and documents.

2.2. Hyperspace Analogue to Language(HAL) (Lund and Burgess, 1996):

HAL builds a word co-occurrence matrix that has both rows and columns representing the words in the vocabulary and the matrix elements are populated with association strength values. The association strength values are calculated by sliding a ”window” the size of which can be varied, over the underlying corpus. The strength of association between the words in the window decreases with the increase in their distance from the focused word. For example, in the sentence ”This is a survey of various semantic similarity measures”, the words ‘survey’ and ‘variety’ have greater association value than the words ‘survey’ and ‘measures.’ Word vectors are formed by taking into consideration both the row and column of the given word. Dimensionality reduction is achieved by removing any columns with low entropy values. The semantic similarity is then calculated by measuring the Euclidean or Manhattan distance between the word vectors.

2.3. Explicit Semantic Analysis (ESA) (Gabrilovich et al., 2007):

ESA measures semantic similarity based on Wiki-pedia concepts. The use of Wikipedia ensures that the proposed method can be used over various domains and languages. Since Wikipedia is constantly updated, the method is adaptable to the changes over time. First, each concept in Wikipedia is represented as an attribute vector of the words that occur in it, then an inverted index is formed, where each word is linked to all the concepts it is associated with. The association strength is weighted using the TF-IDF technique, and the concepts weakly associated with the words are removed. Thus the input text is represented by weighted vectors of concepts called the ”interpretation vectors.” Semantic similarity is measured by calculating the cosine similarity between these word vectors.

2.4. Word-Alignment models (Sultan et al., 2015):

Word-Alignment models calculate the semantic similarity of sentences based on their alignment over a large corpus (Sultan et al., 2014; Kajiwara and Komachi, 2016; Cer et al., 2017). The second, third, and fifth positions in SemEval tasks 2015 were secured by methods based on word alignment. The unsupervised method which was in the fifth place implemented the word alignment technique based on Paraphrase Database (PPDB) (Ganitkevitch et al., 2013). The system calculates the semantic similarity between two sentences as a proportion of the aligned context words in the sentences over the total words in both the sentences. The supervised methods which were at the second and third place used $word2vec$ to obtain the alignment of the words. In the first method, a sentence vector is formed by computing the ”component-wise average” of the words in the sentence, and the cosine similarity between these sentence vectors is used as a measure of semantic similarity. The second supervised method takes into account only those words that have a contextual semantic similarity (Sultan et al., 2015).

2.5. Latent Dirichlet Allocation (LDA) (Sinoara et al., 2019):

LDA is used to represent a topic or the general idea behind a document as a vector rather than every word in the document. This technique is widely used for topic modeling tasks and it has the advantage of reduced dimensionality considering that the topics are significantly less than the actual words in a document (Sinoara et al., 2019). One of the novel approaches to determine document-to-document similarity is the use of vector representation of documents and calculate the cosine similarity between the vectors to ascertain the semantic similarity between documents (Benedetti et al., 2019).

2.6. Normalised Google Distance (Cilibrasi and Vitanyi, 2007):

NGD measures the similarity between two terms based on the results obtained when the terms are queried using the Google search engine. It is based on the assumption that two words occur together more frequently in web-pages if they are more related. Give two terms $t_{1}$ and $t_{2}$ the following formula is used to calculate the NGD between the two terms.

where the functions $f(x)$ and $f(y)$ return the number of hits in Google search of the given terms, $f(x,y)$ returns the number of hits in Google search when the terms are searched together and $G$ represent the total number of pages in the overall google search. NGD is widely used to measure semantic relatedness rather than semantic similarity because related terms occur together more frequently in web pages though they may have opposite meaning.

2.7. Dependency-based models (Agirre et al., 2009):

Dependency-based approaches ascertain the meaning of a given word or phrase using the neighbors of the word within a given window. The dependency-based models initially parse the corpus based on its distribution using Inductive Dependency Parsing (Nivre, 2006). For every given word a ”syntactic context template” is built considering both the nodes preceding and succeeding the word in the built parse tree. For example, the phrase “thinks ¡term¿ delicious” could have a context template as “pizza, burger, food”. Vector representation of a word is formed by adding each window across the location that has the word in consideration, as it’s root word, along with the frequency of the window of words appearing in the entire corpus. Once this vector is formed semantic similarity is calculated using cosine similarity between these vectors. Levy et al. (Levy and Goldberg, 2014a) proposed DEPS embedding as a word-embedding model based on dependency-based bag of words. This model was tested with the WS353 dataset where the task was to rank the similar words above the related words. On plotting a recall precision curve the DEPS curve showed greater affinity towards similarity rankings over BoW methods taken in comparison.

2.8. Kernel-based models (Shawe-Taylor et al., 2004):

Kernel-based methods were used to find patterns in text data thus enabling detecting similarity between text snippets. Two major types of kernels were used in text data namely the string or sequence kernel (Cancedda et al., 2003) and the tree kernel (Moschitti et al., 2008). Moschitti et al. (Moschitti et al., 2008) proposed tree kernels in 2007, that contains three different sub-structures in the tree kernel space namely a subtree - a tree whose root is not a leaf node along with its children nodes, a subset tree - a tree whose root is not a leaf node but not incorporating all its children nodes and does not break the grammatical rules, a partial tree - a tree structure closely similar to subset tree but it doesn’t always follow the grammatical rules. Tree kernels are widely used in identifying a structure in input sentences based on constituency or dependency, taking into consideration the grammatical rules of the language. Kernels are used by machine learning algorithms like Support Vector Machines(SVMs) to adapt to text data in various tasks like Semantic Role Labelling, Paraphrase Identification (Croce et al., 2017), Answer Extraction (Moschitti and Quarteroni, 2008), Question-Answer classification (Moschitti et al., 2007), Relational text categorization (Moschitti, 2008), Answer Re-ranking in QA tasks (Severyn and Moschitti, 2012) and Relational text entailment (Moschitti and Zanzotto, 2007). Severyn et al. (Severyn et al., 2013) proposed a kernel-based semantic similarity method that represents the text directly as “structural objects” using Syntactic tree kernel (Collins and Duffy, 2002b) and Partial tree kernels (Moschitti, 2006). The kernel function then combines the tree structures with semantic feature vectors from two of the best performing models in STS 2012 namely UKP (Bär et al., 2012) and Takelab (Šarić et al., 2012) and some additional features including cosine similarity scores based on named entities, part of speech tags, and so on. The authors compare the performance of the model constructed using four different tree structures namely shallow tree, constituency tree, dependency tree, phrase-dependency tree, and the above-mentioned feature vectors. They establish that the tree kernel models perform better than all feature vectors combined. The model uses Support Vector Regression to obtain the final similarity score and it can be useful in various downstream NLP applications like question-answering, text-entailment extraction, etc. Amir et al. (Amir et al., 2017) proposed another semantic similarity algorithm using kernel functions. They used constituency-based tree kernels where the sentence is broken down into subject, verb, and object based on the assumption most semantic properties of the sentence are attributed to these components. The input sentences are parsed using the Stanford Parser to extract various combinations of subject, verb, and object. The similarity between the various components of the given sentences is calculated using a knowledge base, and different averaging techniques are used to average the similarity values to estimate the overall similarity, and the best among them is chosen based on the root mean squared error value for a particular dataset. In recent research, deep learning methods have been used to replace the traditional machine learning models and efficiently use the structural integrity of kernels in the embedded feature extraction stage (Croce et al., 2017; Collins and Duffy, 2002a). The model which achieved the best results in SemEval-2017 Task 1, proposed by Tian et al. (Tian et al., 2017) uses kernels to extract features from text data to calculate similarity. The model proposed an ensemble model that used both traditional NLP methods and deep learning methods. Two different features are namely the sentence pair matching features and single sentence features were used to predict the similarity values using regressors which added nonlinearity to the prediction. In single sentence feature extraction, dependency-based tree kernels are used to extract the dependency features in one given sentence, and in sentence pair matching features, constituency-based parse tree kernels are used to find the common sub-constructs among the three different characterizations of tree kernel spaces. The final similarity score is accessed by averaging the traditional NLP similarity value and the deep learning-based similarity value. The model achieved a Pearson’s correlation of 73.16% in the STS dataset.

2.9. Word-attention models (Le et al., 2018):

In most of the corpus-based methods all text components are considered to have equal significance; however, human interpretation of measuring similarity usually depends on keywords in a given context. Word attention models capture the importance of the words from underlying corpora (Lopez-Gazpio et al., 2019) before calculating the semantic similarity. Different techniques like word frequency, alignment, word association are used to capture the attention-weights of the text in consideration. Attention Constituency Vector Tree (ACV-Tree) proposed by Le et al. (Le et al., 2018) is similar to a parse tree where one word of a sentence is made the root and the remainder of the sentence is broken as a Noun Phrase (NP) and a Verb Phrase (VP). The nodes in the tree store three different attributes of the word into consideration: the word vector determined by an underlying corpus, the attention-weight, and the ”modification-relations” of the word. The modification relations can be defined as the adjectives or adverbs that modify the meaning of another word. All three components are linked to form the representation of the word. A tree kernel function is used to determine the similarity between two words based on the equation below

where $n_{1},n_{2}$ represent the represents the nodes, $SIM(vec_{1},vec_{2})$ measures the cosine similarity between the vectors, $\delta_{p}(.)$ calculates the number of common subsequences of length $p$, $\lambda$, $\mu$ denote the decay factors for length of the child sequences and the height of the tree respectively, $c_{n_{1}},\mbox{ }c_{n_{2}}$ refer to the children nodes and $l_{m}=min(length(c_{n_{1}}),length(c_{n_{2}}))$. The algorithm is tested using the STS benchmark datasets and has shown better performance in 12 out of 19 chosen STS Datasets (Le et al., 2018; Quan et al., 2019).

Unlike knowledge-based systems, corpus-based systems are language and domain independent (Altınel and Ganiz, 2018). Since they are dependent on statistical measures the methods can be easily adapted across various languages using an effective corpus. With the growth of the internet, building corpora of most languages or domains has become rather easy. Simple web crawling techniques can be used to build large corpora (Baroni et al., 2009). However, the corpus-based methods do not take into consideration the actual meaning of the words. The other challenge faced by corpus-based methods is the need to process the large corpora built, which is a rather time-consuming and resource-dependent task. Since the performance of the algorithms largely depends on the underlying corpus, building an efficient corpus is paramount. Though efforts are made by researchers to build a clean and efficient corpus like the C4 corpus built by web crawling and five steps to clean the corpus (Raffel et al., 2019), an ”ideal corpus” is still not defined by researchers.

Deep Neural Network-based Methods

Semantic similarity methods have exploited the recent developments in neural networks to enhance performance. The most widely used techniques include Convolutional Neural Networks (CNN), Long Short Term Memory (LSTM), Bidirectional Long Short Term Memory (Bi-LSTM), and Recursive Tree LSTM. Deep neural network models are built based on two fundamental operations: convolution and pooling. The convolution operation in text data may be defined as the sum of the element-wise product of a sentence vector and a weight matrix. Convolution operations are used for feature extraction. Pooling operations are used to eliminate features that have a negative impact, and only consider those feature values that have a considerable impact on the task at hand. There are different types of pooling operations and the most widely used is Max pooling, where only the maximum value in the given filter space is selected. This section describes some of the methods that deploy deep neural networks to estimate semantic similarity between text snippets. Although the methods described below exploit word embeddings built using large corpora, deep-neural networks are used to estimate the similarity between the word-embeddings, hence they are classified separately from corpus-based methods.

Wang et al. (Wang et al., 2016) proposed a model to estimate semantic similarity between two sentences based on lexical decomposition and composition. The model uses $word2vec$ pretrained embeddings to form a vector representation of the sentences $s_{1}$ and $s_{2}$. A similarity matrix $M$ of dimension $i$ x $j$ is built where i and j are the number of words in sentence 1 ($S_{1}$) and sentence 2 ($S_{2}$) respectively. The cells of the matrix are populated with the cosine similarity between the words in the indices of the matrix. Three different functions are used to construct semantic matching vectors $\vec{s_{1}}$ and $\vec{s_{2}}$ , the global, local, and max function. The global function constructs the semantic matching vector of $S_{1}$ by taking the weighted sum of the vectors, of all the words in $S_{2}$, the local function, takes into consideration only word vectors within a given window size, and the max function takes only the vectors of the words, that have the maximum similarity. The second phase of the algorithm uses three different decomposition functions - rigid, linear, and orthogonal - to estimate the similarity component and the dissimilarity component between the sentence vectors and the semantic matching vectors. Both the similarity component and the dissimilarity component vectors are passed through a two-channel convolution layer followed by a single max-pooling layer. The similarity is then calculated using a sigmoid layer that estimates the similarity value within the range of 0 and 1. The model was tested using the QASent dataset (Wang et al., 2007) and the WikiQA dataset (Meek et al., 2018). The two measures used to estimate the performance are mean average precision (MAP) and mean reciprocal rank (MRR). The model achieves the best MAP in the QASent dataset and the best MAP and MRR in the WikiQA dataset. Yang Shao (Shao, 2017) proposed a semantic similarity algorithm that exploits, the recent development in neural networks using $GloVe$ word embeddings. Given two sentences, the model predicts a probability distribution over set semantic similarity values. The pre-processing steps involve the removal of punctuation, tokenization, and using $GloVe$ vectors to replace words with word embeddings. The length of the input is set to 30 words, which is achieved by removal or padding as deemed necessary. Some special hand-crafted features like flag values indicating if the words or numbers occurred in both the sentences and POS tagging one hot encoded values, were added to the $GloVe$ vectors. The vectors are then fed to a CNN with 300 filters and one max-pooling layer which is used to form the sentence vectors. ReLU activation function is used in the convolution layer. The semantic difference between the vectors is calculated by the element-wise absolute difference and the element-wise multiplication of the two, sentence-vectors generated. The vectors are further passed through two fully-connected layers, which predicts the probability distribution of the semantic similarity values. The model performance was evaluated using the SemEval datasets where the model was ranked 3rd in SemEval 2017 dataset track.

The LSTM networks are a special kind of Recurrent Neural Networks (RNN). While processing text data, it is essential for the networks to remember previous words, to capture the context, and RNNs have the capacity to do so. However, not all the previous content has significance over the next word/phrase, hence RNNs suffer the drawback of long term dependency. LSTMs are designed to overcome this problem. LSTMs have gates which enable the network to choose the content it has to remember. For example, consider the text snippet, “Mary is from Finland. She is fluent in Finnish. She loves to travel.” While we reach the second sentence of the text snippet, it is essential to remember the words “Mary” and “Finland.” However, on reaching the third sentence the network may forget the word “Finland.” The architecture of LSTMs allows this. Many researchers use the LSTM architecture to measure semantic similarity between blocks of text. Tien et al. (Tien et al., 2019) uses a network combined with LSTM and CNN to form a sentence embedding from pretrained word embeddings followed by an LSTM architecture to predict their similarity. Tai et al. (Tai et al., 2015) proposed an LSTM architecture to estimate the semantic similarity between two given sentences. Initially, the sentences are converted to sentence representations using Tree-LSTM over the parse tree of the sentences. These sentence representations are then, fed to a neural network that calculates the absolute distance between the vectors and the angle between the vectors. The experiment was conducted using the SICK dataset, and the similarity measure varies with the range 1 to 5. The hidden layer consisted of 50 neurons and the final softmax layer classifies the sentences over the given range. The Tree-LSTM model achieved better Pearson’s and Spearman’s correlation in the gold standard datasets, than the other neural network models in comparison.

He and Lin (He and Lin, 2016) proposed a hybrid architecture using Bi-LSTM and CNN to estimate the semantic similarity of the model. Bi-LSTMs have two LSTMs that run parallel, one from the beginning of the sentence and one from the end, thus capturing the entire context. In their model, He and Lin use Bi-LSTM for context modelling. A pairwise word interaction model is built that calculates a comparison unit between the vectors derived from the hidden states of the two LSTMs using the below formula

where $\vec{h_{1}}$ and $\vec{h_{2}}$ represent the vectors from the hidden state of the LSTMs and the functions $cos()$, $euc()$, $manh()$ calculate the Cosine distance, Euclidean distance, and Manhattan distance, respectively. This model is similar to other recent neural network-based word attention models (Bahdanau et al., 2015; Alexander M. Rush et al., 2015). However, attention weights are not added, rather the distances are added as weights. The word interaction model is followed by a similarity focus layer where weights are added to the word interactions (calculated in the previous layers) based on their importance in determining the similarity. These re-weighted vectors are fed to the final convolution network. The network is composed of alternating spatial convolution layers and spatial max pooling layers, ReLU activation function is used and at the network ends with two fully connected layers followed by a LogSoftmax layer to obtain a non-linear solution. This model outperforms the previously mentioned Tree-LSTM model on the SICK dataset.

Lopez-Gazpio et al. (Lopez-Gazpio et al., 2019) proposed an extension to the existing Decomposable Attention Model (DAM) proposed by Parikh et al. (Parikh et al., 2016) which was originally used for Natural Language Inference(NLI). NLI is used to categorize a given text block to a particular relation like entailment, neutral, or contradiction. The DAM model used feed-forward neural networks in three consecutive layers the attention layer, comparison layer, and aggregation layer. Given two sentences the attention layer produces two attention vectors for each sentence by finding the overlap between them. The comparison layer concatenates the attention vectors with the sentence vectors to form a single representative vector for each sentence. The final aggregation layer flattens the vectors and calculates the probability distribution over the given values. Lopez-Gazpio et al. (Lopez-Gazpio et al., 2019) used word n-grams to capture attention in the first layer instead of individual words. $n-grams$ maybe defined as a sequence of n words that are contiguous with the given word, n-grams are used to capture the context in various NLP tasks. In order to accommodate n-grams, a Recurrent Neural Network (RNN) is added to the attention layer. Variations were proposed by replacing RNN with Long-Term Short memory (LSTM) and Convolutional Neural Network (CNN). The model was used for semantic similarity calculations by replacing the final classes of entailment relationships with semantic similarity ranges from 0 to 5. The models achieved better performance in capturing the semantic similarity in the SICK dataset and the STS benchmark dataset when compared to DAM and other models like Sent2vec (Pagliardini et al., 2018) and BiLSTM among others.

Transformer-based models: Vaswani et al. (Vaswani et al., 2017) proposed a transformer model that relies on attention mechanisms to capture the semantic properties of words in the embeddings. The transformer has two parts ‘encoder’ and ‘decoder’. The encoder consists of layers of multi-head attention mechanisms followed by a fully connected feed-forward neural network. The decoder is similar to the encoder with one additional layer of multi-head attention which captures the attention weights in the output of the encoder. Although this model was proposed for the machine translation task, Devlin et al. (Devlin et al., 2019) used the transformer model to generate BERT word embeddings. Sun et al. (Sun et al., 2020) proposed a multi-tasking framework using transformers called ERNIE 2.0. In this framework, the model is continuously pretrained i.e., when a new task is presented the model is fine-tuned to accommodate the new task while retaining the previously gained knowledge. The model outperformed BERT. XLNet proposed by Yang et al. (Yang et al., 2019) used an autoregression model as opposed to the autoencoder model and outperformed BERT and ERNIE 2.0. A number of variations of BERT models were proposed based on the corpus used to train the model and by optimizing the computational resources. Lan et al. (Lan et al., 2019) proposed ALBERT, with two techniques to reduce the computational complexity of BERT namely ‘factorized embedding parameterization’ and ‘cross-layer parameter sharing’. ALBERT outperformed all the above three models. Other variations of BERT models that use transformers include TinyBERT (Jiao et al., 2019), RoBERTa (Liu et al., 2019; Sanh et al., 2019), and a domain-specific variation trained on a scientific corpus with a focus on the BioMedical domain the SciBERT (Beltagy et al., 2019). Raffel et al. (Raffel et al., 2019) proposed a transformer model with a well-defined corpus called ‘Colossal Clean Crawled Corpus’ or C4 to train the model named T5-11B. Unlike BERT they adopt a ‘text-to-text framework’ where the input sequence is attached with a token to identify the NLP task to be performed thus eliminating the two stages pre-training and fine-tuning. They propose five different versions of their model based on the number of trainable parameters each model has namely 1) T5-Small 2) T5-Base 3) T5-Large 4) T5-3B and 5)T511B and they have 60 million, 220 million, 770 million, 3 billion, and 11 billion parameters respectively. This model outperformed all other transformer-based models and achieved the state of the art results. As a result of their study, they confirm that the performance of the models increases with increased data and computational power and the performance can be further improved if larger models are built and it is important to note that in order to replicate their best model five GPUs are required among other resources. A compilation of the various transformer-based models and their Pearson’s correlation on the STS-B dataset is provided below in Table LABEL:tab:berttable.

Deep neural network-based methods outperform most of the traditional methods and the recent success of transformer-based models have served as a breakthrough in semantic similarity research. However, implementation of deep-learning models requires large computational resources, though variations of the models to minimize the computational resources are being proposed we see that the performance of the model takes a hit as well, for example, TinyBERT (Jiao et al., 2019). And the performance of the models is largely increased by the use of a bigger corpus which again poses the challenge of building an ideal corpus. Most deep-learning models are ”black-box” models and it is difficult to ascertain the features based on which the performance is achieved, hence it becomes difficult to be interpreted unlike in the case of corpus-based methods that have a strong mathematical foundation. Various fields like finance, insurance, etc., that deal with sensitive data may be reluctant to deploy deep neural network-based methods due to their lack of interpretability.

Hybrid Methods

Based on all the previously discussed methods we see that each has its advantages and disadvantages. The knowledge-based methods exploit the underlying ontologies to disambiguate synonyms, while corpus-based methods are versatile as they can be used across languages. Deep neural network-based systems, though computationally expensive, provide better results. However, many researchers have found ways to exploit the best of each method and build hybrid models to measure semantic similarity. In this section, we describe the methodologies used in some of the widely used hybrid models.

Novel Approach to a Semantically-Aware Representation of Items (NASARI) (Camacho-Collados et al., 2015): Camacho Collados et al. (Camacho-Collados et al., 2015) proposed an approach the $NASARI$ where the knowledge source BabelNet is used to build a corpus based on which vector representation for concepts (words or group of words) are formed. Initially, the Wikipedia pages associated with a given concept, in this case, the synset of BabelNet, and all the outgoing links from the given page are used to form a sub-corpus for the specific concept. The sub-corpus is further expanded with the Wikipedia pages of the hypernyms and hyponyms of the concept in the BabelNet network. The entire Wikipedia is considered as the reference corpus. Two different types of vector representation were proposed. In the first method, weighted vectors were formed using lexical specificity. Lexical specificity is a statistical method of identifying the most representative words for a given text, based on the hypergeometric distribution (sampling without replacement). Let ”$T$ and $t$” denote the total content words in the reference corpus $RC$ and sub-corpus $SC$ respectively and ”$F$ and $f$” denote the frequency of the given word in the reference corpus $RC$ and sub-corpus $SC$ respectively, then lexical specificity can be represented by the below equation

X represents a random variable that follows a hypergeometric relation with the parameters $T$, $t$ and $F$ and $P(X\geq f)$ is defined as,

$P(X=i)$ is the probability of a given term appearing exactly $i$ times in the given sub-corpus in hypergeometric distribution with $T$, $t$ and $F$. The second method forms a cluster of words in the sub-corpus that share a common hypernym in the WordNet taxonomy which is embedded in BabelNet. The specificity is then measured based on the frequency of the hypernym and all its hyponyms in the taxonomy, even those that did not occur in the given sub-corpus. This clustering technique forms a unified representation of the words that preserve the semantic properties. The specificity values are added as weights in both methods to rank the terms in a given text. The first method of vector representation was called $NASARI_{lexical}$ and the second method was called $NASARI_{unified}$. The similarity between these vectors is calculated using the measure called Weighted Overlap (Pilehvar et al., 2013) as,

where $O$ denotes the overlapping terms in each vector and $rank(d,\vec{v_{i}})$ represent the rank of the term $d$ in the vector $v_{i}$.

Camacho Collados et al. (Camacho-Collados et al., 2016) proposed an extension to their previous work and proposed a third vector representation by mapping the lexical vector to the semantic space of word embeddings produced by complex word embedding techniques like $word2vec$. This representation was called as $NASARI_{embedded}$. The similarity is measured as the cosine similarity between these vectors. All three methods were tested across the gold standard datasets M&C, WS-Sim and SimLex-999. $NASARI_{lexical}$ achieved higher Pearson’s and Spearman’s correlation in average over the three datasets in comparison with other methods like ESA, $word2vec$, and $lin$.

Most Suitable Sense Annotation (MSSA) (Ruas et al., 2019): Ruas et al. proposed three different methodologies to form word-sense embeddings. Given a corpus, the word-sense disambiguation step is performed using one of the three proposed methods: Most Suitable Sense Annotation (MSSA), Most Suitable Sense Annotation N Refined (MSSA-NR), and Most Suitable Sense Annotation Dijkstra (MSSA-D). Given a corpus each word in the corpus is associated with a synset in the WordNet ontology and ”gloss-average-vector” is calculated for each synset. The gloss-average-vector is formed using the vector representation of the words in the gloss of each synset. MSSA calculates the gloss-average-vector using a small window of words and returns the synset of the word which has the highest gloss-average-vector value. MSSA-D, however, considers the entire document from the first word to the last word and then determines the associated synset. These two systems use Google News vectorshttps://code.google.com/archive/p/word2vec/ . to form the synset-embeddings. MSSA-NR is an iterative model, where the first pass produces the synset-embeddings, that are fed back in the second pass as a replacement to gloss-average-vectors to produce more refined synset-embeddings. These synset-embeddings are then fed to a $word2vec$ CBOW model to produce multi-sense word embeddings that are used to calculate the semantic similarity. This combination of MSSA variations and $word2vec$ produced solid results in gold standard datasets like R&G, M&C, WS353-Sim, and SimLex-999 (Ruas et al., 2019).

Unsupervised Ensemble Semantic Textual Similarity Methods (UESTS) (Hassan et al., 2019): Hassan et al. proposed an ensemble semantic similarity method based on an underlying unsupervised word-aligner. The model calculates the semantic similarity as the weighted sum of four different semantic similarity measures between sentences $S_{1}$ and $S_{2}$ using the equation below

$sim_{WAL}(S_{1},S_{2})$ calculates similarity using a synset-based word aligner. The similarity between text is measured based on the number of shared neighbors each term has in the BableNet taxonomy. $sim_{SC}(S_{1},S_{2})$ measures similarity using soft cardinality measure between the terms in comparison. The soft cardinality function treats each word as a set and the similarity between them as an intersection between the sets. $sim_{embed}(S_{1},S_{2})$ forms word vector representations using the word embeddings proposed by Baroni et al. (Baroni et al., 2014). Then similarity is measured as the cosine value between the two vectors. $sim_{ED}(S_{1},S_{2})$ is a measure of dissimilarity between two given sentences. The edit distance is defined as the minimum number of edits it takes to convert one sentence to another. The edits may involve insertion, deletion, or substitution. $sim_{ED}(S_{1},S_{2})$ uses word-sense edit distance where word-senses are taken into consideration instead of actual words themselves. The hyperparameters $\alpha$, $\beta$, $\gamma$, and $\theta$ were tuned to values between 0 and 0.5 for different STS benchmark datasets. The ensemble model outperformed the STS benchmark unsupervised models in the 2017 SemEval series on various STS benchmark datasets.

Hybrid methods exploit both the structural efficiency offered by knowledge-based methods and the versatility of corpus-based methods. Many studies have been conducted to build multi-sense embeddings in order to incorporate the actual meaning of words into word vectors. Iacobacci et al. formed word embeddings called ”Sensembed” by using BabelNet to form a sense annotated corpus and then using $word2vec$ to build word vectors thus having different vectors for different senses of the words. As we can see, hybrid models compensate for the shortcomings of one method by incorporating other methods. Hence the performance of hybrid methods is comparatively high. The first 5 places of SemEval 2017 semantic similarity tasks were awarded to ensemble models which clearly shows the shift in research towards hybrid models (Cer et al., 2017).

Analysis of Survey

This section discusses the method used to build this survey article and provides an overview of the various research articles taken into consideration.

The articles considered for this survey were obtained using the Google Scholar search engine and the keywords used include “semantic similarity, word embedding, knowledge-based methods, corpus-based methods, deep neural network-based semantic similarity, LSTM, text processing, and semantic similarity datasets.” The results of the search were fine-tuned using various parameters like the Journal Ranking, Google Scholar Index, number of citations, year of publication, etc. Only articles published in journals with Scimago Journal ranking of Quartile 1 and conferences that have a Google metrics H-index above 50 were considered. Exceptions were made for some articles that have a higher impact and relevance. The table of references sorted by the year of publication is included in Appendix B as Table LABEL:tab:my-table. The table records 1) Title, 2) Year of Publication, 3) Author Names, 4) Venue, 5) SJR Quartile (for journals), 6) H-Index, and 7) Number of Citations (as of 02.04.2020). Some of the statistical results of the chosen articles are shown in the figures below. These figures highlight the quality of the articles chosen that in turn, highlights the quality of the survey. Figure 3 shows the distribution of the referenced articles over conferences, journals, and others. 55% of the articles are from conferences and 38% of the articles are from journals. The remaining 7% of the articles are from arXiv and books. However, they have rather a high impact in relation to the topic of the survey. Figure 3 highlights the distribution of the selected articles over the years. Nearly 72% of the chosen articles are works carried out after 2010, the remaining 28% of the articles represent the traditional methods adopted during the early stages of the evolution of semantic similarity. Figure 5 represents the citation range of the articles. 54% of the articles have 50 to 500 citations, 26% have 1,000-5,000 citations, and 12% of the article have more than 5000 citations. We see that 33% of the articles have citations below 50 however, all these articles are published after the year 2017 which accounts for the fewer citations.

2. Word-cloud generation:

We implemented a simple python code to generate a word cloud using the abstracts from all the articles used in this survey. The abstracts from all the 118 articles were used to build a dataset that was then used in the python code. The extracted abstracts are initially pre-processed by converting the text to lower case, removing the punctuation, and removing the most commonly used English stop words available in the nltkhttp://www.nltk.org/. library. Then the word-cloud is built using the $wordcloud$ python library. The word cloud thus built is shown in Figure LABEL:fig:word_cloud. From the word cloud, we infer that though different keywords were used in our search for articles the general focus of the selected articles is semantic similarity. In a word cloud, the size of the words is proportional to the frequency of use of these words. The word “word” is considerably bigger than the word “sentence” showing that most of the research works focus on word-to-word similarity rather than sentence-to-sentence similarity. We could also infer that the words ”vector” and ”representation” have been used more frequently than the words ”information”, ”context”, and ”concept” indicating the influence of corpus-based methods over knowledge-based methods. With the given word cloud we showcase the focus of the survey graphically.

Conclusion

Measuring semantic similarity between two text snippets has been one of the most challenging tasks in the field of Natural Language Processing. Various methodologies have been proposed over the years to measure semantic similarity and this survey discusses the evolution, advantages, and disadvantages of these methods. Knowledge-based methods taken into consideration the actual meaning of text however, they are not adaptable across different domains and languages. Corpus-based methods have a statistical background and can be implemented across languages but they do not take into consideration the actual meaning of the text. Deep neural network-based methods show better performance, but they require high computational resources and lack interpretability. Hybrid methods are formed to take advantage of the benefits from different methods compensating for the shortcomings of each other. It is clear from the survey that each method has its advantages and disadvantages and it is difficult to choose one best model, however, most recent hybrid methods have shown promising results over other independent models. While the focus of recent research is shifted towards building more semantically aware word embeddings, and the transformer models have shown promising results, the need for determining a balance between computational efficiency and performance is still a work in progress. Research gaps can also be seen in areas such as building domain-specific word embeddings, addressing the need for an ideal corpus. This survey would serve as a good foundation for researchers who intend to find new methods to measure semantic similarity.

References

Appendix A Semantic distance measures and their formulae

Appendix B Table of References