WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domain

Introduction

Efficient financial markets incorporate all price relevant information available to investors at that point of time. Unstructured data, such as textual data, help complement structured data traditionally used by investors. For example, in addition to quantitative data such as firm’s financial performance, the tone and sentiment of firms’ financial reports, earnings calls and social media posts can also influence the stock price movement Bochkay et al. (2020). We aim to capture these textual features with the help of pre-trained deep learning models, which have shown superior performance in a variety of Natural Language Processing (NLP) tasks Radford et al. (2019); Devlin et al. (2018); Liu et al. (2019); Lewis et al. (2020). However, the language used in finance and economics is likely to be different from the language of common usage. A statement like “The crude oil prices are going up” has a negative sentiment for the financial markets, but it does not contain traditionally negative words such as danger, hate, fear, etc. Loughran and McDonald (2011). Therefore, it is necessary to develop a domain-specific language model training methodology that improves the performance in the downstream NLP tasks like managers’ sentiment analysis and financial news classification.

Previous research, for example, Yang et al. (2020); Araci (2019) have pre-trained the state-of-the-art language models like BERT Devlin et al. (2018) with financial documents, but suffer from two major limitations. First, financial domain knowledge and adaptation are not utilized in the pre-training process. We argue that the financial terminologies play a critical role in understanding the language used in financial markets, and expect a performance improvement after incorporating the financial domain knowledge into the pre-training process. Second, the lack of different evaluation benchmarks limit the test the language models’ performance in finance-related tasks.

In this work, we propose a simple yet effective language model pre-training methodology with preferential token masking and prediction of phrases. This helps capture the fact that many financial terms are actually multi-token phrases, such as margin call and break-even analysis. We contribute and make public two language models trained using this technique. Financial LANGuage Model (FLANG-BERT) is based on BERT-base architecture Devlin et al. (2018), which has a relatively small memory footprint and inference time. It also enables comparison with previous works, most of which are based on BERT. We also contribute FLANG-ELECTRA, our best performing model, based on the ELECTRA-base architecture Clark et al. (2020), where we introduce a span boundary objective on the ELECTRA generator pre-training task to learn robust financial multi-word representations while masking contiguous spans of text. We show that FLANG-BERT outperforms all previous works in nearly all our benchmarks, and FLANG-ELECTRA further improves the performance giving two new state-of-the-art models. Our training methodology can be extended to other domains that would benefit from domain adaptation.

Financial domain benchmarks are critical to evaluate the newly developed financial language models. Inspired by GLUE Wang et al. (2018), a set of comprehensive benchmarks across multiple NLP tasks, we construct Financial Language Understanding Evaluation (FLUE) benchmarks. FLUE consists of 5 financial domain tasks: financial sentiment analysis, news headline classification, name entity recognition, structure boundary detection, and question answering. We intend for this benchmark suite to be a standard for evaluation of natural language tasks in financial domain, subject to appropriate license and privacy considerations. All proposed benchmarks will be made publicly available on Github and Huggingface.

We propose masking finance-specific words and phrase masking for pre-training language model, as well as a span boundary objective to build robust multi-word representations.

We contribute finance-related benchmarks with 5 NLP tasks: financial sentiment analysis, news headline classification, named entity recognition, structure boundary detection, question answering. This results in a comprehensive suite of finance benchmarks, with licensing details in Table 1.

We make all our models and code publicly available, for easier development and further research by the NLP and Finance community. Specifically, we contribute FLANG-BERT and FLANG-ELECTRA language models, and all the benchmarks in FLUE.

Related Work

Language models pre-trained on unlabeled textual data, such as BERT Devlin et al. (2018), ELMo Peters et al. (2018) and ROBERTA Liu et al. (2019), have significantly improved the state-of-the-art in many natural language tasks. Newer models introduce different training objectives: BART Lewis et al. (2020) uses denoising auto-encoder objective for sequence-to-sequence pre-training; Span-BERT Joshi et al. (2019) uses a pre-training methodology that predicts spans of text; ELECTRA Clark et al. (2020) uses token detection for training, where it corrupts some tokens using a generator network and predicts if the tokens are corrupted using a discriminator.

Most language models use Masked Language Modeling (MLM) Devlin et al. (2018) as a training objective. It typically involves randomly masking a percentage of tokens in a text, and using surrounding text to predict the masked tokens. A variety of masking techniques have been used for domain-specific pre-training. While some works Glass et al. (2020); Sun et al. (2019b) propose rule based masking strategies that work better than random masking, other works Kang et al. (2020) attempt to find optimal masking policy automatically using techniques such as reinforcement learning.

While the models trained on general English language perform well, domain-specific pre-training can further increase the performance on a particular domain of text Sun et al. (2019a); Gururangan et al. (2020). For example, BioBERT Lee et al. (2019) on biomedical domain, ClinicalBERT Alsentzer et al. (2019) on clinical domain, SciBERT Beltagy et al. (2019) on scientific publications domain, etc. There have been some works on financial domain as well: previous works by Araci (2019); Yang et al. (2020) directly fine-tune BERT trained on financial corpus for sentiment analysis and question answering tasks respectively. FinBERT Liu et al. (2020) uses multi-task pre-training to improve performance. The previous works in financial domain rely on basic architectures/ training schemes and do not use finance-specific knowledge. Furthermore, FinBERT is pre-trained with the objective of optimizing performance for sentiment analysis, while we build a generalizable model performing well on a diverse set of tasks. We use and demonstrate that finance specific knowledge and vocabulary can further improve the performance of the model.

Wang et al. (2018) created General Language Understudy Evaluation(GLUE), a collection of benchmark tasks for training, evaluating, and analyzing language model designed for non-domain specific tasks. For financial domain, the benchmark suite isn’t as exhaustive. Malo et al. (2014) created Financial PhraseBank dataset for Sentiment analysis classification. Maia et al. (2018) created two tasks in FiQA FiQA 2018: Task-1 for Sentiment Analysis Regression and Task 2 dataset for Question Answering task in finance. Other datasets include gold news headline datase Sinha and Khandait (2020), financial NER Alvarado et al. (2015) and Structure Boundary Detection FinSBD3 (2021). Recent financial language models Araci (2019); Yang et al. (2020) evaluate their efficacy only on sentiment analysis tasks. We use datasets from existing literature and create a set of heterogeneous benchmark tasks FLUE (Financial Language Understanding Evaluation) for better comprehensive evaluation.

Benchmarks (FLUE) and Datasets

We introduce Financial Language Understanding Evaluation (FLUE), a set of comprehensive benchmarks across 5 financial tasks. The statistics for FLUE are summarized in Table 1 along with the licensing details for public use. All FLUE benchmark datasets have low ethical risks and do not expose any sensitive information of any organization/ individual. Additionally, we have obtained approval for the authors of each dataset for this FLUE benchmark.

Serving as a fundamental task for textual analysis, this task received a lot of attention in finance domain (Loughran and McDonald, 2011; Garcia, 2013). In our FLUE benchmark, we include both sentiment analysis tasks: regression and classification. For classification, we use Financial PhraseBank dataset Malo et al. (2014) which provides the sentiment labels annotated by humans for financial news sequences. For regression, we use FiQA 2018 task-1 (Aspect-based financial sentiment analysis) dataset Maia et al. (2018), which contains both headlines and microblogs.

1.2 News Headline Classification

The financial phrases contain information on multiple dimensions other than the sentiment. Financial news headlines contain important time sensitive information on price changes. To explore our model on those dimensions, we use the Gold news headline dataset created by Sinha and Khandait (2020). The dataset is a collection of 11,412 news headlines, with 9 binary labels.

1.3 Named Entity Recognition

Name entity recognition (NER) is key task to analysing any financial text as it can be used along with the Knowledge Graphs to better understand interdependence of different financial entities linked through location, organisation and person. Given a text, NER can identify and classify tokens into specified categories such as person, organisation, location and miscellaneous. We use dataset released by Alvarado et al. (2015) for NER task on financial domain text.

1.4 Structure Boundary Detection

Boundary detection of different structure is fundamental challenge in processing text data. Here we employ the dataset shared in the task FinSBD-3 of FinSBD3 (2021)FinWeb-2021 workshop. The goal of the task is to find the boundaries of different components of text (sentences, lists and list items, including structure elements like footer, header, tables). We chose this dataset as it not only identifies boundaries of sentences but also identifies boundaries of other structural elements.

1.5 Question Answering

Question answering system which can answer the finance domain question is essential to any digital assistant. To evaluate our language model’s ability on QA task we employ the dataset ("Opinion-based QA over financial data") released in FiQA FiQA 2018 open challenge Task 2 Maia et al. (2018).

2 Pre-training Datasets

For pre-training, we use a mix of general English language datasets and finance specific datasets. For English, we use BooksCorpus Zhu et al. (2015) (800M words) and English Wikipedia (2500M words). For the domain specific datasets, we use six publicly available datasets, they are: 1) SEC 10-K and 10-Q financial reports, 2) Earning Conference calls, 3) Analyst Reports, 4) Reuters Financial News, 5) Bloomberg Financial News, and 6) Investopedia. The details for these datasets are summarized in Table 13 and a brief description of each dataset is given in the Appendix Section 7.1.

Model

For FLANG-BERT, we add financial word and phrase masking, while for FLANG-ELECTRA, we also add a span boundary objective. The addition of financial word and phrasal masking is model agnostic and can be used for any model with a generator.

Previous works Liu et al. (2020); Yang et al. (2020); Araci (2019) on financial language modeling use MLM objective for pre-training, which masks some tokens randomly and uses the prediction of those tokens as a training objective. However, there is empirical evidence Sun et al. (2019b); Kang et al. (2020); Glass et al. (2020) that masking some words strategically which carry more information improves performance on downstream tasks.

Hence, we propose masking financial words preferentially. To this end, we use Investopedia Financial Term Dictionary Investopedia to create a comprehensive financial dictionary, which lists the commonly used technical terms in financial markets and literature. We expand our list by adding words/phrases from other financial vocabulary lists available online Vocabulary.com ; MyVocabulary.com ; TheStreet .

Our dictionary contains more than 8200 words and phrases. For preferential masking, we mask the single word financial tokens with a probability 30% and randomly mask other tokens with 70% percent probability. Like original BERT pre-training scheme, we mask a cumulative total of 15% of all tokens, such that the total number of tokens being masked in each round is same as the original BERT pre-training approach. Table 10 shows that masking financial terms with a 30% probability gives the lowest perplexity score when pre-training either BERT and ELECTRA with additional vocabulary.

2 Phrase Masking

Many financial terms are phrases with multiple tokens. It has been shown Sun et al. (2019b); Joshi et al. (2019) that masking phrases instead of words could leads to better learning of the phrase content. Building on that, we use phrase-based masking in the language model. We perform a two-phase training: in the first phase, we only use word masking to mask single tokens and train the language model; in the second phase, we add phrase masking.

For a financial term of token length $n$, we mask it with a probability of 30%. We replace all tokens in a financial phrase with a single [MASK] token. We add all the financial phrases in the model vocabulary and predict the phrase with the usual masked language modeling objective.

3 Span Boundary Objective

We add the Span Boundary Objective to the loss function along with the MLM loss in the pre-training stage, in addition to the word and the phrasal level masking and the modified vocabulary. Our final loss has three parts:

is the Maximum Likelihood Loss of the ELECTRA generator (G). We also modify the token masking to randomly mask contiguous spans from a geometric distribution of length L $\sim$ Geo(p), which is skewed towards smaller spans. We follow the results of Joshi et al. (2019) and set p = 0.2.

This loss term is the standard ELECTRA implementation. $L_{Disc}$ penalizes if the discriminator detects a token generated by the generator as replaced when it is a non-corrupt token or if the token generated by G is corrupt and the discriminator detects it as original.

This term penalizes the low probability of a token being generated given span boundaries (the the representations of tokens present before and after the masked contiguous span). The position of the left boundary token is $x_{start-1}$ and the position of the right boundary token is $x_{end+1}$. By looking at words before and after spans and then trying to generate the tokens in the span, this term helps the model to build multi-word representations of financial terms that are not captured in our vocabulary.

Here the function $f(c\dot{)}$ is the representation function for the $i^{th}$ token in the span and is defined by two feed forward layers:

Our model is then pre-trained and optimized based on this combined loss function.

4 Contrastive Loss for Fine-tuning

While most language models are fine-tuned for supervised classification by using cross-entropy loss Devlin et al. (2018); Liu et al. (2019); Radford et al. (2019), we use additional supervised contrastive learning loss for fine-tuning for classification Gunel et al. (2021). This loss function captures the similarities between examples of the same class and contrasts them with the examples from other classes. Details about Supervised Contrastive Loss are given in Appendix Section 7.3. Here, we only add this loss to the fine-tuning of Financial Phrasebank Dataset and the Headlines Dataset as shown in Tables 5 and 6.

Experiments

All experiments were conducted with PyTorch Paszke et al. (2019) on NVIDIA V100 GPUs. We initialized each model with their respective pre-trained version on the Huggingface’s Transformers library Wolf et al. (2020). We further pre-trained each model for 4 more epochs on the training data. We used 2 epochs with only single token masking and the later 2 epochs for both word and phrase masking. Using this multi-stage setup gives the lowest model perplexity as shown in Table 11.

We used ELECTRA-base pre-trained model as our base architecture. ELECTRA corrupts the input by replacing tokens with words sampled from a generator and trains a discriminator model that predicts whether each token in the corrupted input was replaced by a generator sample. This enables it to learn from all input tokens rather than just masked out tokens and is a good fit for our preferential masking approach. We compare our results the following models:

BERT-base and ELECTRA-base: We use the BERT-base model Devlin et al. (2018) and the ELECTRA-base model Clark et al. (2020) from Huggingface Wolf et al. (2020) and fine-tuned it directly for our tasks.

finBERT Yang et al. (2020): We used finBERT model and fine-tune on our tasks.

FLANG-BERT (ours) (Financial LANGuage Model based on BERT): For direct comparison with finBERT, we use our method to train a BERT-base model on our training corpus in a multi-stage manner (Table 11), masking single tokens from financial vocabulary in the first stage and then masking both words and phrases in the second stage.

ELECTRA w/ AD (Additional Data): The ELECTRA base model pre-trained on our financial training corpus.

ELECTRA w/ AD + FV (Financial Vocabulary): The ELECTRA Base model is pre-trained on our training corpus, while masking single tokens from financial vocabulary with a higher probability.

ELECTRA w/ AD + PFV (Phrase Financial Vocabulary). The ELECTRA Base model pre-trained on our training corpus in a multi-stage manner (Table 11), masking only single-word tokens from financial vocabulary in the first stage and masking both words and phrases in the second stage.

FLANG-ELECTRA (Financial LANGuage Model based on ELECTRA): ELECTRA w/ AD + PFV (Phrase Financial Vocabulary) + SBO (Span Boundary Objective). It is pre-trained on our training corpus in the described multi-stage manner with the span boundary and in-filling training objective.

ELECTRA w/ AD + PFV + SBO + SCL (Contrastive Loss): We use our final language model (FLANG-ELECTRA) but add a contrastive loss term to fine-tune on supervised classification tasks.

2 Benchmark Results

Summarized results on all benchmarks of our model and baselines are shown in Table 2.

The results of sentiment classification on Financial Phrase Bank sentiment dataset are shown in Table 2. From the accuracy numbers listed in the Table 2, it is evident that FLANG-BERT improves hugely on performance of FinBERT and our final language model(FLANG-ELECTRA) significantly outperforms all the baseline models on the sentiment classification task on the Financial Phrase Bank dataset, achieving state of the art results. Results in Table 5 highlight the importance of each step in our experiment setup described in Section 5.1. As the previous state of art performance on this dataset is already in the higher 80s, we use an additional metric: marginal increase in performance over FinBERT ($\Delta MP$) to demonstrate our techniques. We calculate ($\Delta MP$) as given in equation 1:

where the Metric is Accuracy for the Financial Phrasebank Dataset and is F1 score for News Headlines Dataset.

2.2 FiQA Sentiment Regression

The results of sentiment regression analysis on the FiQA dataset are shown in Table 3. Evaluation of models is done on two regression evaluation measures Mean Squared Error (MSE) and R Square (R2). Our transformer based architectures outperform conventional techniques like SC-V and RCNN. FLANG-BERT model achieves significant improvement on both BERT and finBERT and FLANG-ELECTRA outperforms all models and achieves state of art result for the sentiment regression analysis task on the FIQA dataset.

2.3 News Headline Classification

The results of news headline classification for 9 binary classification tasks on Gold headline dataset are shown in Table 6. All the deep learning based language models perform much better than Support Vector Machines. Our ELECTRA-based language model (FLANG-ELECTRA) achieves the highest mean F-1 score compared to other language models. FLANG-BERT performs better than BERT, which again highlights the importance of our setup.

2.4 Named Entity Recognition

The results of NER on financial NER dataset provided by Alvarado et al. (2015) are shown in Table 4. The margin of improvement is more muted in this benchmark. Our models outperform the baselines in a multi-token setting. The multi-token setting refers to all tokens in a word being set to the same label when a word is split into multiple tokens, instead of only labeling the first token and ignoring the rest. Our hypothesis is that when the task doesn’t require domain specific knowledge, like NER, pre-training language model on domain specific data does not help.

2.5 Structure Boundary Detection

The results of structure boundary detection task on FinSBD3 dataset from FinSBD3 (2021)FinWeb-2021 are shown in Table 7. In this table, note that the "Special Tokens" setting refers to adding special tokens that are commonly used by pre-trained transformers such as [CLS] to the input. Our models perform similarly or slightly better to baseline architectures. This could be because SBD, like NER, relies more on language cues rather than finance keywords for inference and further gives evidence to the hypothesis that when the task doesn’t require domain specific knowledge, one should not get improvement by pre-training a language model on domain specific data. However, our model still performs significantly better than FinBERT.

2.6 Question Answering

On Question-Answering, our models outperform the previous works, as shown in Table 8. For evaluation, we compare the following metrics Michael and Joseph : Precision, nDCG—A higher value means that more relevant documents are retrieved first, and MRR—A higher value means that the first relevant item is retrieved earlier. FLANG-BERT, FLANG-ELECTRA outperform other models on all metrics by a huge margin, but do not outperform SpanBERT pre-trained with Additional Data.

3 Ablation Studies

We conduct multiple ablation studies to understand the individual impact of our techniques on performance. Our studies in Table 10 show that preferentially masking 30% of the financial tokens gives the least perplexity for each model. Furthermore, we find that using single-word financial terminologies in the first two pre-training epochs and multi-word terminologies in the next two gives the lowest perplexity score (Table 11). Table 9 shows that the use of additional data and domain specific preferential masking give substantial increase in performance for our FLUE tasks. Addition of the Span Boundary Objective on the ELECTRA generator gives the best performing model when compared to other similar encoder based architectures like SpanBERT, DistilBERT and BERT. In Table 12, we also show that pre-training models using our methodology gives the lowest perplexity scores when compared to prior baselines. The details for the studies can be found in Table 9 and Appendix Section 7.2.

4 Discussion

In conclusion, both FLANG-ELECTRA and FLANG-BERT outperform the base architectures (ELECTRA and BERT, respectively). FLANG-BERT also outperforms FinBERT on all the benchmarks, with the same number of parameters. Additionally, on relatively domain-agnostic tasks such as Named Entity Recognition, the improvements are muted. The performance is hugely improved in tasks which utilize finance specific language, such as sentiment analysis, sentence classification and question answering. Overall, the dramatic improvement in most benchmarks suggests that our technique yields state-of-the-art financial language models. We also note that our vocabulary based preferential masking training methodology is both architecture and domain independent and can be generalized to other language models and domains.

Conclusion

We contribute two language models in the finance domain, which use domain-specific word and phrase masking as a pre-training objective. Additionally, we contribute a comprehensive suite of benchmarks in finance domain across 5 natural language tasks, including new benchmarks using public sources. Our language model outperforms previous language models on all the benchmarks. We will release our models, code and benchmark data on acceptance. We also note that our method is not specific to finance and can be used for any domain-specific language model training.

Acknowledgements

We would like to thank the anonymous reviewers for their comments. We appreciate the generous support of Azure credits from Microsoft made available for this research via the Georgia institute of Technology Cloud Hub. This work is supported in part by the J.P. Morgan AI Faculty Research Award. Any opinions, findings, and conclusions in this paper are those of the authors only and do not necessarily reflect the views of the sponsors.

Ethics Statement

We give full credit to the respective authors of each dataset included in our FLUE benchmark and have obtained their permissions for the inclusion of each dataset in FLUE. All FLUE benchmark datasets have low ethical risks and do not expose any sensitive or personal identifiable information. We also obtain explicit permissions to use the datasets given in section 13 for pre-training of the FLANG models from the respective sources.

We understand that training large language models has big carbon-footprint and we have tried to minimize the number of full-scale pre-training runs. The addition of preferential masking and the span boundary objective have minimal computation overhead when compared to pre-training traditional BERT/ELECTRA. We hope that future models work towards lower carbon footprint to reduce the environment costs of pre-training for more sustainable and ethical AI.

Limitations

Some limitations to our work are: 1) We have not included abstractive generation or summarization tasks in the FLUE benchmark, due to a lack of large, annotated datasets. Future work can be directed towards summarization efforts for the financial domain. 2) We do not include social media data like twitter and reddit in our pre-training step, despite the heavy impact of social media on some financial markets like crypto currencies. This is because of the informal usage of textual data which impedes the formal and syntactical correctness of most financial documents. 3) The models are trained and tested on English tasks and may not perform well on non-English text. The limited availability of non-English domain specific vocabulary makes building multi-lingual FLANG models difficult. 4) While the methodologies presented in this paper can work well for any similarly structured domain like clinical data, it is often difficult to obtain a vocabulary term lists and dictionaries for certain domains. 5) We limit ourselves to using encoder based architectures due to the nature of the popular financial domain specific tasks. Future works can explore the use of other models like GPT3 and T5 for the domain.

References

Appendix

Table 13 summarizes the financial datasets used for pre-training. It also presents the percentage of each dataset sampled in one training epoch. A brief description of each dataset used for pre-training is given below:

Most U.S. public firms are required by the U.S. Securities and Exchange Commission (SEC) to submit annual report (10-K) and quarterly report (10-Q), to provide detailed information about the firm’s business, risk factors, and financial performance. 10-K and 10-Q filings were analyzed in (Li, 2010; Loughran and McDonald, 2011; Buehlmaier and Whited, 2018; Chava and Paradkar, 2016). We download the 10-K and 10-Q filings from SEC EDGAR during 1993–2020.

1.2 Earnings Conference Calls

The earnings conference calls are held by public companies to covey critical corporate information to the investors and analysts (Bushee et al., 2003; Bowen et al., 2002). SeekingAlpha, as a crowd-sourced website in the United States, provides investing information for a large number of public companies and publishes textual transcripts of many earnings conference calls. Bochkay et al. (2020) use the earnings conference call transcripts to analyze the stock market response to the language extremity. Chava et al. (2019) use BERT to construct emerging technology related discussions in earnings calls and evaluate whether it is just hype. Chava et al. (2020) employ RoBERTa to extract environmental related discussion in earnings calls and analyze whether managers walk their talk. We collect 151,359 earnings call transcripts from SeekingAlpha from Jan. 2000 to Jul. 2019. Chava et al. (2022) use BERT to construct a text-based firm-level inflation exposure measure on earning call transcripts.

1.3 Analyst Reports

Security analysts generate reports related to a firms’ future performance after collecting and analyzing the relevant information. Most analyst reports contains earnings forecast, stock recommendation, and stock price target (Asquith et al., 2005). We collect around 201 analyst reports on public firms from LexisNexis. This corpus contains the language the analysts use to disseminate the new information and their interpretation of previous released information to the investors.

1.4 Reuters Financial News

Financial news corpus is helpful in analyzing the language used in business society. The Thomson Reuters Text Research Collection (TRC2) contains over 1.8M financial news stories during 2008–2009, which is deployed in prior literature (Araci, 2019). We use 10% of this corpus to pre-train our model.

1.5 Bloomberg Financial News

Bloomberg disseminates business and market news to the market investors. We obtain the publicly available Bloomberg news articles provided by Philippe Remy (2015), which is used in Ding et al. (2014) to predict the return of Standard & Poor’s 500 stock (S&P 500) index.

1.6 Investopedia

Investopedia is a financial website which serves as a comprehensive financial dictionary and provides definition and explanation for financial terminologies used in business world. We download the 638 articles for the financial concepts, and use them to pre-train our model. These articles not only provide definitions of financial terms, but also show how they are interrelated to each other.

2 Ablation Studies

For the first study, we try different configurations while preferentially masking financial terms in the pre-training. Table 10 shows the impact of masking different percentages of Financial Terms on the model perplexity. The perplexities are calculated while keeping the total percentage of masked tokens for all vocabulary at 15 percent. Table 10 shows that masking 30 percent of financial terms gives the least perplexity on the validation set. We also experiment with the multi-stage masking, where in the first stage (first n epochs) we use only the single-word financial tokens and in the second stage (next m epochs) we use both: word and phrasal financial vocabulary masking. Table 11 shows that masking single-word financial vocabulary in the first 2 epochs and masking all financial terms has the lowest perplexity score.

2.2 Perplexity on Validation Set

For the second study, we compute perplexity of the language model on the validation set after pre-training. We report the perplexity scores in Table 12. We notice that FLANG-BERT significantly lowers the perplexity on validation set, relative to BERT and FinBERT Araci (2019). Despite all models having the same number of parameters, ELECTRA based models show lower perplexity scores. For ablation study, we keep ELECTRA architecture fixed and notice that pre-training with financial data along with general English data lowers perplexity compared to base ELECTRA. Further reduction is seen when using our token masking approach with financial keywords, suggesting that domain specific masking is helpful for domain specific language models. Pre-training with phrase based masking with the span boundary objective in the generator stage results in the best performance, validating the performance of our technique.

2.3 FPB Sentiment Classification

For the third study, we fine-tune the models for sentiment analysis on the Financial PhraseBank Dataset Malo et al. (2014) and report the accuracy in Table 5. We perform a detailed ablation study on ELECTRA architectures with our various techniques. The results suggest that pre-training on financial data improves accuracy from 88.1% to 91.1%, and using a financial vocabulary for token masking further improves the performance to 91.4%. Span boundary objective is even more effective, improving accuracy to $>91.5\%$. Using contrastive learning for fine-tuning further enables an accuracy of 92.1%, which is significantly higher than previous works.

3 Supervised Contrastive Loss

Language models are usually fine-tuned Devlin et al. (2018); Liu et al. (2019); Radford et al. (2019) for supervised classification tasks by using cross entropy loss $L_{CE}$:

where $N$ is the number of samples, $C$ is the number of classes, $(x_{i},y_{i})$ are the sentence and label pairs for sample $i$ and $\hat{y}_{i,c}$ is the model output for probability of sample $i$ having class $c$.

Gunel et al. (2021) showed that using an additional supervised contrastive learning loss $L_{SCL}$ for fine-tuning pre-trained language models improves performance. The loss is meant to capture the similarities between examples of the same class and contrast them with the examples from other classes:

where $N_{c}$ is the number of samples of class $c$.

where $\lambda$ is a variable for weighing the two losses.