NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation

Introduction

Data augmentation, the act of creating new datapoints by slightly modifying copies or creating synthetic data based on existing data, is an important component in the robustness evaluation of models in natural language processing (NLP) and in enhancing the diversity of their training data. Most data augmentation techniques create examples through transformations of existing examples which are based on prior task-specific knowledge Feng et al. (2021); Chen et al. (2021). Such transformations seek to disrupt model predictions or can be used as training candidates for improving regularization and denoising models, for example, through consistency training Xie et al. (2020); Figure 1 shows a few possible transformations for a sample sentence.

However, a vast majority of transformations do not alter the structure of examples in drastic and meaningful ways, rendering them qualitatively less effective as potential training or test examples. Moreover, different NLP tasks may benefit from transforming different linguistic properties. for example, changing the word “happy” to “very happy” in an input is more relevant for sentiment analysis than for summarization Mille et al. (2021). Despite this, many transformations may be universially useful, for example changing places to ones from different geographic regions, or changing names to those from different cultures. As such, having a single place to collect both task-specific and task-independent augmentations will ease the barrier to creating appropriate suites of augmentations that should be applied to different tasks.

Natural language and its long-tailed nature (Bamman, 2017) lead to a very high diversity of possible surface forms. If only a handful number of ways to paraphrase a text are available, it can be hard to generalize across radically different surface forms. Besides, data drawn i.i.d. from such long-tailed distribution represents itself exactly in proportion to its occurence in the dataset. Evaluating NLP systems on such data means that the head of the distribution is emphasized even in the test dataset and that rare phenomena are implicitly ignored. However, informed transformations or the identification of such tail examples may require a wide range of domain knowledge or specific cultural backgrounds. We thus argue that a collection of transformations that should be applied to NLP datasets should be done by capitalizing on the “wisdom-of-researchers”.

To enable more diverse and better characterized data during testing and training, we create a Python-based natural language augmentation framework, NL-Augmenterhttps://github.com/GEM-benchmark/NL-Augmenter. With the help of researchers across subfields in computational linguistics and NLP, we collect many creative ways to augment data for natural language tasks. To encourage task-specific implementations, we tie each transformation to a widely-used data format (e.g. text pair, a question-answer pair, etc.) along with various task types (e.g. entailment, tagging, etc.) that they intend to benefit. We demonstrate the efficacy of NL-Augmenter by using some of its transformations to analyze the robustness of popular natural language models.

A majority of the augmentations that the framework supports are transformations of single sentences that aim to paraphrase these sentences in various ways. NL-Augmenter loosens the definition of “transformations” from the logic-centric view of strict equivalence to the more descriptive view of linguistics, closely resembling Bhagat and Hovy (2013)’s “quasi-paraphrases”. We extend this to accommodate noise, intentional and accidental human mistakes, sociolinguistic variation, semantically-valid style, syntax changes, as well as artificial constructs that are unambiguous to humans (Tan et al., 2021b). Some transformations vary the socio-linguistic perspective permitting a crucial source of variation wherein language goals span beyond conveying ideas and content.

In addition to transformations, NL-Augmenter also provides a variety of filters, which can be used to filter data and create subpopulations of given inputs, according to features such as input complexity, input size, etc. Unlike a transformation, the output of a filter is a boolean value, indicating whether the input meets the filter criterion, e.g. whether the input text is toxic. The filters allow splitting existing datasets and hence evaluating models on subsets with specific linguistic properties.

In this paper, we apply the collected transformations and filters to several datasets and show to what exent the different types of perturbations do affect some models.

The paper is organized as follows. We first discuss the rise of participatory benchmarks in Section 2. In Section 3, we introduce the participatory workshop and the repository of NL-Augmenter. In Section 4, we present the robustness analysis performed on the participants’ submissions, and in Section 4 we provide a broader impact discussion. All filters and transformations are listed and described in details in the Appendix.

Related Work

NL-Augmenter enables both data augmentation and robustness testing by constructing the library in a participatory fashion. We provide an overview of the related work in these lines of research.

To address the problem of under-resourced African languages in machine translation, Masakhane adopted a bottom-up, participatory approach to construct machine translation benchmarks for over thirty languages (Nekoto et al., 2020). This collaborative approach is increasingly adopted in the NLP communtiy to create evolving benchmarks in response to the rapid pace of NLP progress. The Generation Evaluation and Metrics benchmark (Gehrmann et al., 2021), which started the development of NL-Augmenter, is a participatory project to document and improve tasks and their evaluation in natural language generation. BIG-Benchhttps://github.com/google/BIG-bench proposes a collaborative framework to collect a large suite of few-shot tasks to gauge the abilities of large, pretrained language models. DynaBench (Kiela et al., 2021) proposes to iteratively evaluate models in a human-in-the-loop fashion by constructing more challenging examples after each round of model evaluation. SyntaxGym Gauthier et al. (2020) provides a standardized platform for researchers to contribute and use evaluation sets and focuses on targeted syntactic evaluation of Language Models (LMs), particularly psycholinguistically motivated ones. In contrast, our transformations operate on a larger variety of tasks and model types — they are not required to be syntactically or even linguistically motivated.

2 Wisdom-of-Researchers

There are many ways to introduce variation in a sentence without altering its meaning; the lived experiences of a diverse group of individuals could help with identifying and codifying the myriad dimensions of variation as executable transformations (Tan et al., 2021b). Leveraging the wisdom-of-the-crowd Galton (1907); Yi et al. (2010) is common in our field of natural language processing, with the use of Amazon Mechanical Turk to generate and annotate data in exchange for monetary returns. The aforementioned BIG-Bench project, hosted on GitHub, offers co-authorship in exchange for task contribution. We can think of this as a sort of wisdom-of-researchers. Similarly, we crowd-source transformations, in the form of Python code snippets, in return for co-authorship.

3 Robustness Evaluation Tools

There are many projects with similar goals that inspired NL-Augmenter. For example Gardner et al. (2020) proposed creating “contrast” sets of perturbed test examples. In their approach, each example is manually perturbed, which may lead to higher-quality results but is costly to replicate for each new task due to scale and annotator cost. TextAttack (Morris et al., 2020) is a library enabling the adversarial evaluation of English NLP models. Partially overcoming this limitation, TextFlint (Wang et al., 2021a), supports robustness evaluation in English and Chinese. It covers linguistic and task-specficic transformations, adversarial attacks, and subpopulation analyses. In contrast, while the majority are focused on English, NL-Augmenter comprises transformations and filters that work for many different languages and each contribution can specify a set of supported languages.

Robustness Gym (Goel et al., 2021) unifies four different types of robustness tests — subpopulations, transformations, adversarial attacks, and evaluation sets — under a single interface. However, it depends on existing libraries for its transformations. In contrast, NL-Augmenter focuses on compiling a set of transformations in an open source and collaborative fashion, which is reflected in its size and diversity. Checklist (Ribeiro et al., 2020) argues for the need to go beyond simple accuracy and evaluate the model on basic linguistic capabilities, for example their response to negations. Polyjuice (Wu et al., 2021) perturbs examples using GPT-2 — though this is automatic and scalable, it offers limited control over type of challenging examples generated, making fine-grained analysis beyond global challenge-set level difficult. In contrast, our method offers a richer taxonomy with $117$ (and growing) transformations for extensive analysis and comparison.

Tan et al. (2021b) propose decomposing each real world environment into a set of dimensions before using randomly sampled and adversarially optimized transformations to measure the model’s average- and worst-case performance along each dimension. NL-Augmenter can be used, out-of-the-box, to measure average-case performance and we plan to extend it to support worst-case evaluation.

NL-Augmenter

NL-Augmenter is a crowd-sourced suite to facilitate rapid augmentation of data for NLP tasks to assist in training and evaluating models. NL-augmenter was introduced in Mille et al. (2021) in the context of the creation of evaluation suites for the GEM benchmark (Gehrmann et al., 2021, 2022); three types of evaluation sets were proposed: (i) transformations, i.e. original test sets are perturbed in different ways (e.g. backtranslation, introduction of typographical errors, etc.), (ii) subpopulations, i.e. test subsets filtered according to features such as input complexity, input size, etc.; and (iii) data shifts, i.e. new test sets that do not contain any of the original test set material.

In this paper, we present a participant-driven repository for creating and testing transformations and filters, and for applying them to all dataset splits (training, development, evaluation) and to all NLP tasks (NLG, labeling, question answering, etc.). As shown by Mille et al. (2021), applying filters and tranformations to development/evaluation data splits allows for testing the robustness of models and for identifying possible biases; on the other hand, applying transformations and filters to training data (data augmentation) allows for possibly mitigating the detected robustness and bias issues Wang et al. (2021b); Pruksachatkun et al. (2021); Si et al. (2021).

In this section, we provide organizational details, list the transformations and filters that the repository currently contains, and we present the list of tags we associated to tranformations and filters and how we introduced them.

A workshop was organized towards constructing this full-fledged participant-driven repository. Unlike a traditional workshop wherein people submit papers, participants were asked to submit python implementations of transformations to the GitHub repository. Organizers of this workshop created a base repository extending Mille et al. (2021)’s NLG evaluation suite and incorporated a set of interfaces, each of which catered to popular NL example formats. This formed the backbone of the repository. A sample set of transformations and filters alongwith evaluation scripts were provided as starter code. Figure 2 show an annotated code snippet of a submission. Following the format of BIG-Bench’s review process, multiple review criteria were designed for accepting contributions. The review criteria (see Appendix A) guided participants to follow a style guide, incorporate test cases in JSON format, and encouraged novelty and specificity. Apart from the general software development advantages of test cases, they made reviewing simpler by providing an overview of the transformation’s capability and scope of generations.

2 Transformations and filters

Tables 1 and 2 list respectively the 117 transformations and 23 filters that are currently found in the NL-Augmenter repository (alphabetically ordered according to the submission name in the repository). For each transformation/filter, a link to the corresponding Appendix subsection is provided, where a detailed description, illustrations and an exernal link to implementation in the NL-Augmenter repository can be found.

3 Tags for the classification of perturbations

We defined a list of tags which are useful for an efficient navigation in the pool of existing perturbations and for understanding the performance characteristics of the contributed transformations and filters (see e.g. the robustness analysis presented in Section 4.2). There are three main categories of tags: (i) General properties tags, (ii) Output properties tags, and (iii) Processing properties tags.

General properties tags are shown in Table 3, and cover the type of the augmentation, i.e. whether it is a transformation or a filter (Augmented set type), its general purpose, i.e. whether it is intended for augmentation, robustness, etc. (General purpose), for which NLP tasks the created data will be useful (Task type), to which languages it has been applied (Language(s)), and on which linguistic level of representation it operates, i.e. semantic, syntactic, lexical, etc. (Linguistic level).

Output properties tags, shown in Table 4, apply to transformations only; they provide indications about how the data was affected during the respective transformations. There are currently six properties in this category: one to capture the number of different outputs that a transformation can produce (Output/Input ratio), one to capture in which aspect the input and the output are alike (Input/Ouptut similarity), and four to capture intrinsic qualities of the produced text or structured data, namely how were the meaning, the grammaticality, the readability and the naturalness affected by the transformation (respectively Meaning preservation, Grammaticality preservation, Readability preservation and Naturalness preservation). Note that apart from Output/Input ratio, the output properties tags need to be specified manually for each transformation/filter (see Section 3.4), and are thus subject to the interpretation of the annotator.

Processing properties tags, shown in Table 5, capture information related to the type of processing applied on the input (Input data processing), the type of algorithm used (Algorithm type), how it is implemented (Implementation), its estimated precision and recall (Precision/recall) and computational complexity (Computational complexity / Time), and whether an accelerator is required to apply the transformation/filter (GPU required?).

4 Tag retrieval and assignment

Transformation and filters are assigned tags for each of the properties listed in Tables 3-5. There are two sources for the tags: (i) assigning them manually, and (ii) using existing metadata embedded in the respective source code implementations of each given transformation and filter. The in-code metadata (see e.g. the Keywords field in Figure 2) provides descriptions for each one identificable aspects such as the language(s) supported, the type of task that the transformation or filter is applicable for, and other characteristical keywords. The specification and type of this metadata was pre-defined as a requirement for all contributors to the NL-Augmenter project to enable identification of the type of transformation of filter being written by their respective author(s).

This metadata was initally collected through the creation of an automated script which programatically iterated through each transformation and filter and gathered all stated metadata. The metadata was then mapped by the script into discrete property groups as defined in Tables 3-5. All contributing authors were invited to review the initally collected metadata and, where possible, add additional data.

Robustness Analysis

All authors of the accepted perturbations were asked to provide the task performance scores for each of their respective transformations or filters. In Section 4.1 we provide details on how the scores were obtained, and in Section 4.2 we provide a first analysis of these scores.

The perturbations are currently split into three groups, according to the task(s) they will be evaluated on: text classification tasks, tagging tasks, and question-answering tasks. For experiments in this paper, we focus on text classification and on the relevant perturbations. We compare the models’ performance on the original data and on the perturbed data. The percentage of sentences being changed by a transformation (transformation rate) and the percentage of performance drop on the perturbed data compared to the performance on the original data (score variation) are reported.

Tasks. We choose four evaluation datasets among three English NLP tasks: (1) sentiment analysis on both short sentences (SST-2 Socher et al. (2013)) and full paragraphs (IMDB Movie Review Maas et al. (2011)), (2) Duplicate question detection (QQP) Wang et al. (2019a), and (3) Natural Language Inference (MNLI) Williams et al. (2017). These tasks cover both classifications on single sentences, as well as pairwise comparisons, and have been widely used in various counterfactual analysis and augmentation experiments Wu et al. (2021); Kaushik et al. (2019); Gardner et al. (2020); Ribeiro et al. (2020).

Evaluation models. We represent each dataset/task with its corresponding most downloaded large model hosted on Huggingface Wolf et al. (2020), resulting in four models for evaluation: roberta-base-SST-2, roberta-base-imdb, roberta-large-mnli, and bert-base-uncased-QQP.huggingface.co/{ textattack/roberta-base-SST-2, textattack/roberta-base-imdb, textattack/bert-base-uncased-QQP, roberta-large-mnli }

Perturbation strategy. For each task, we perturb a random sample of 20% of the validation set. Since all the transformations are on single text snippets, for datasets with sentence pairs, i.e., QQP and MNLI, we perturb the first question and the premise sentence, respectively.

2 Results and Analysis

In this section, Tables 6 to 16 show the results of the robustness analysis performed on the four datasets described in Section 4.1 and presented according to the tags introduced in Section 3.3.

General purpose (Table 6): Transformations designed with a “robustness testing” objective displayed mean performance drops between 9% and 13.7% across models. Interestingly, 34 sentence transformations designed for “augmentation” tasks showed similar mean robustness drops ranging between 4% and 13%, emphasizing the need to draw on the paraphrasing literature to improve robustness testing.

Task type (Table 7): The results table shows that there is not necessarily a correlation between which task a transformation is marked to be relevant for and which task it actually challenges the robustness of the models on.

Linguistic level (Table 8): Transformations making character level and morphological changes were able to show drastic levels of drops in performance compared to those making lexical or syntactic changes. These drops in performance were consistent across all four models. roberta-large finetuned on the MNLI dataset was the most brittle - character-level transformations on an average dropped performance by over 31% and morphological changes dropped it by 28% while those which made lexical changes displayed a mean drop of 4.4%. The visual_attack_letters (B.111) transformation, which replaces characters with similarly looking ones (like y and v), shows a large accuracy drop from 94% to 56% on the ‘roberta-base‘ model fine tuned on SST. ‘bert-base-uncased‘ fine-tuned on the QQP dataset drops from 92 to 69. roberta-large-mnli drops from 91 to 47. In the case of visual_attack_letters, one can easily conceive a scenario in which a model is applied to OCR text which likely exhibit similar properties. In this case, one may expect similarly poor performance, arguably attributed to a narrow set of characters that the models have been exposed to.

Meaning preservation (Table 10): 22 transformations which were marked as highly meaning preserving surprisingly showed a larger average performance drop as compared to 20 of those which were marked as possibly meaning changing. Not discounting the possibility of the noisiness of the transformation’s logic, we believe further investigation could help understand whether models focus on the meaning of words or sentences or take shortcuts by focusing on commonly occuring surface forms associated with a particular prediction, as was already shown for some phenomena by McCoy et al. (2019), among others.

Grammaticality preservation (Table 11): Preserving grammaticality did not correlate with high robustness. Transformations marked as grammaticality always-preserved showed significant average drops of 10.6%, 8.1% and 4.6% across roberta-base-SST-2, roberta-large-mnli and bert-base-uncased-QQP respectively. For example, the grapheme_to_phoneme transformation showed drastic drops in performance: 13%, 20% and 13% respectively.

Readability and Naturalness (Tables 12-13): In general, as expected, the transformations tagged as modifying the readability or naturalness show large drops across all tasks and models, in particular the ones tagged as “always imparing” the input.

Unsurprisingly, many of the injected perturbations, despite being artificial would not distract human readers from the actual meaning and intent of the text (e.g. simple_ciphers transformation (B.91)). Character level perturbations might not distract human readers as much as compared to word level perturbations but the above language models on the other hand behaved contrarily. Such departure from learning meaningful abstractions is further validated with the low correlation of grammaticality preservation and robustness. These results further re-question how we can expand these models from being just pure statistical learners to those which can incorporate meaning and surface-level abstraction, both across natural as well as artificial constructs. The large drops in performance of such perturbations necessitate looking at expanding training sets with even artificial data sources as well expand our definitions of text similarity from pure linguistic ones to those which abstract morphological, visual and other errors which can be unambiguous to humans.

Tables 9, 14, 15 and 16 show the robustness scores for Input/Output similarity, Input processing, Implementation and Algorithm type respectively. The score drops for these criteria may not be easily interpretable; e.g. that model-based implementations showed comparatively larger average drops as compared to rule-based implementations may not be due to the difference in implementation, but rather to which transformations were implemented that way .

Discussion and Broader Impact

In Section 4.2, we analyze the results of applying some of the transformations on existing datasets and running models on the perturbed data. Even though it was not possible to test all of the currently existing perturbations (mostly due to time constraints), the overall results show that the tested perturbations do pose a challenge to different models on different tasks, with quasi-systematic score drops. However, with so many transformations applied to four different datasets, the presented robustness analysis can only be shallow, and a separate analysis of each transformation would be needed in order to get more informative insights. Second, our superficial analysis above relies on tags which were in many cases annotated by hand, and some of the suprising results (e.g. meaning-preserving transformations are more challenging than non-meaning-preserving ones) may reflect a lack of consistency in the annotations. We believe that assessing the quality of the tag assignment so as to ensure a high inter-annotator agreement will be needed for reliable analyses in the future. Finally, the current robustness analysis only shows that the perturbations are effective for detecting a possible weakness in a model; further experiments are needed to demonstrate that the perturbations can also help mitigating the weaknesses they bring to light.

Dilution of Contributions

While this is not our intent, there is a risk in large scale collections of work like this that individual contributions are being less appreciated than releasing them as a standalone project. This risk is a tradeoff with the advantage that it becomes much easier to switch between different transformations, which can lead to a better adoption of introduced methods. To proactively give appropriate credit, each transformation has a data card mentioning the contributors and all participants are listed as co-authors of this paper. We further encourage all users of our repository to cite the work that a specific implementation builds on, if appropriate. The relevant citations are listed on the respective data cards and in the description in the appendix. In the same vein, there is a risk of NL-Augmenter as a whole to monopolize the augmentation space due to its large scope, leading to less usage of related work which may cover additional transformations or filters. While this is not our intention and we actively worked with contributors to related repositories to integrate their work, we encourage researchers to try other solutions as well.

Participatory Setup

Conducting research in environments with a shared mission, a low barrier of entry, and directly involving affected communities was popularizied by Nekoto et al. (2020). This kind of participatory work has many advantages, most notably that it changes the typically prescriptive research workflow toward a more inclusive one. Another advantage is that through open science, anyone can help shape the overall mission and improve the end result. Following the related BIG-bench Srivastava et al. (2022) project, we aimed to design NL-Augmenter in a similar spirit – by providing the infrastructure, the participation barrier is reduced to filling a templated interface and providing test example. By making the interface as flexible as possible, the contributions range from filters for subpopulations with specific protected attributes to transformations via neural style transfer. Through this wide range, we hope that researchers can apply a wider range of augmentation and evaluations strategies to their data and models.

Conclusion

In this paper, we introduced NL-Augmenter, a framework for text transformations and filters with the goal to assisst in robustness testing and other data augmentation tasks. We demonstrated that through an open participation strategy, NL-Augmenter can cover a much wider set of languages, tasks, transformations, and filters than related work without a loss of focus. In total, our repository provides $>$ 117 transformations and $>$ 23 filters which are all documented and tested and will contribute toward more robust NLP models and an evaluation thereof. As we point out in our analysis, there is much room to improve NL-Augmenter. We welcome future contributions to improve its coverage of the potential augmentation space and to address its current shortcomings. Future work may further include data augmentation experiments at a larger scale to investigate the effect on model robustness.

Organization

NL-Augmenter is a large effort organized by researchers and developers ranging across different niches of NLP. To acknowledge everyone’s contributions, we list the contribution statements below for all.

Kaustubh Dhole, Varun Gangal, Sebastian Gehrmann, Aadesh Gupta, Zhenhao Li, Saad Mahmood, Simon Mille, Jascha Sohl-Dickstein, Ashish Shrivastava, Samson Tan, Tongshuang Wu and Abinaya Mahendiran make up the steering committee. Jinho D. Choi, Eduard Hovy & Sebastian Ruder provided guidance and feedback. Kaustubh Dhole coordinates and leads the NL-Augmenter effort. All others provide feedback and discuss larger decisions regarding the direction of NL-Augmenter and act as organizers and reviewers.

Repository:

Kaustubh, Aadesh, Zhenhao, Tongshuang, Ashish, Saad, Varun & Abinaya created the interfaces and the base repository NL-Augmenter for participants to contribute. This was also a continuation of the repository developed for creating challenge sets Mille et al. (2021) for GEM Gehrmann et al. (2021). All the other authors expanded this repository with their implementations.

Reviewers:

Kaustubh, Simon, Zhenhao, Sebastian, Varun, Samson, Abinaya, Saad, Tongshuang, Aadesh, Ondrej were involved in reviewing the submissions of participants of the first phase. In the 2nd phase, all other authors performed a cross-review, in which participants were paired with 3 other partcipants. This was followed by a meta review by the organizers.

Robustness Evaluation:

Ashish, Tongshuang, Kaustubh & Zhenhao created the evaluation engine. Simon, Kaustubh, Saad, Abinaya & Tongshuang performed the robustness analysis.

Website:

Aadesh and Sebastian created the webpages for the project.

References

Appendix A Review criteria for submission evaluation

Figure 3 shows the detailed review criteria used for evaluating the transformation and filters submissions.

Appendix B Transformations

The following is the list of all accepted transformations to NL-Augmenter project. Many of the transformations tokenize the sentences using SpaCyhttps://spacy.io/ or NLTK Bird (2006) tokenizers. We discuss the implementations of each alongwith their limitations. The title of each transformation subsection is clickable and redirects to the actual python implementation. Many of the transformations use external libraries and we urge readers to look at each implementation and its corresponding ‘requirements.txt‘ files.

This transformation replaces a word or phrase with its abbreviated counterpart “homework” -> “hwk” using a web-scraped slang dictionary.Scraped from https://www.noslang.com/dictionary

🖙 You $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ yu driving at 80 miles per hour $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ mph is why insurance is $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ tis so freaking $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ friggin expensive.

B.2 Add Hash-Tags

This transformation uses words in the text to generate hashtags. These hastags are then appeneded to the original text. Using the same words appearing in the sentence to generate the hashtags acts as redundant noise that models should learn to ignore. Hashtags are widespread in social media channels and are used to draw attention to the source text and also as a quick stylistic device.

🖙 I love domino’s pizza. $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ #LovePizza #Love #I #Pizza

B.3 Adjectives Antonyms Switch

This transformation switches English adjectives in a sentence with their WordNet Miller (1998) antonyms to generate new sentences with possibly different meanings and can be useful for tasks like Paraphrase Detection, Paraphrase Generation, Semantic Similarity, and Recognizing Textual Entailment.

🖙 Amanda’s mother was very beautiful $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ ugly .

B.4 AmericanizeBritishizeEnglish

This transformation takes a sentence and tries to convert it from British English to American English and vice-versa. A select set of words have been taken from hyperreality@GitHub .

🖙 I love the pastel colours $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ colors

B.5 AntonymsSubstitute

This transformation introduces semantic diversity by replacing an even number of adjective/adverb antonyms in a given text. We assume that an even number of antonyms transforms will revert back sentence semantics; however, an odd number of transforms will revert the semantics. Thus, our transform only applies to the sentence that has an even number of revertible adjectives or adverbs. We called this mechanism double negation.

🖙 Steve is able $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ unable to recommend movies that depicts the lives of beautiful $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ ugly minds.

B.6 Auxiliary Negation Removal

This is a low-coverage transformation which targets sentences that contain negations. It removes negations in English auxillaries and attempts to generate new sentences with the oppposite meaning.

🖙 Ujjal Dev Dosanjh was not $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ Ujjal Dev Dosanjh was the 1st Premier of British Columbia from 1871 to 1872.

B.7 AzertyQwertyCharsSwap

🖙 Preferably use the above download link, as the release tarballs are generated deterministically $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ qre generqted deterministicqlly whereas GitHub’s are not.

B.8 BackTranslation

This transformation translates a given English sentence into German and back to English.This transformation acts like a light paraphraser. Multiple variations can be easily created via changing parameters like the language as well as the translation models which are available in plenty. Backtranslation has been quite popular now and has been a quick way to augment examples Li and Specia (2019); Sugiyama and Yoshinaga (2019).

🖙 Andrew finally returned $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ eventually gave Chris the French book the French book I bought last week.

B.9 BackTranslation for Named Entity Recognition

This transformation splits the token sequences into segments of entity mention(s) and “contexts” around the entity mention(s). Backtranslation is used to paraphrase the contexts around the entity mention(s), thus resulting in a different surface form from the original token sequence. The resultant tokens are also assigned new tags. Exploiting this transformation has shown to empirically benefit named entity tagging Yaseen and Langer (2021) and hence could arguably benefit other low-resource tagging tasks Bhatt and Dhole (2020); Khachatrian et al. (2019); Gupta et al. (2021).

B.10 Butter Fingers Perturbation

This perturbation adds noise to all types of text sources (sentence, paragraph, etc.) proportional to noise erupting from keyboard typos making common spelling errors. Few letters picked at random are replaced with letters which are at keyboard positions near the source letter. The implementation has been borrowed from here Yorke as used in Mille et al. (2021). There has also been some recent work in NoiseQA Ravichander et al. (2021) to mimick keyboard typos.

🖙 Sentences $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ Senhences with gapping, such as Paul likes coffee $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ coffwe and Mary tea, lack an overt predicate to indicate $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ indicatx the relation $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ relauion between two or more arguments $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ argumentd .

B.11 Butter Fingers Perturbation For Indian Languages

This implements the butter fingers perturbation as used above for 7 Indian languages: Bangla, Gujarati, Hindi, Kannada, Malayalam, Oriya, Punjabi, Tamil, and Telugu. The implementation considers the InScript keyboard https://en.wikipedia.org/wiki/InScript_keyboard which is decreed as a standard for Indian scripts.

B.12 Change Character Case

This transformation acts like a perturbation and randomly swaps the casing of some of the letters. The transformation’s outputs will not work with uncased models or languages without casing.

🖙 Alice in Wonderland is a 2010 American live- action $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ actIon / animated $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ anImated dark fantasy $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ faNtasy adventure film.

B.13 Change Date Format

This transformation changes the format of dates.

🖙 The first known case of COVID-19 was identified in Wuhan, China in December $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ Dec 2019.

B.14 Change Person Named Entities

This perturbation changes the name of the person from one name to another by making use of the lexicon of person names in Ribeiro et al. (2020).

🖙 Andrew $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ Nathaniel finally returned the French book to Chris that I bought last week

B.15 Change Two Way Named Entities

This perturbation also changes the name of the person but also makes a parallel change in the label or reference text with the same name making it useful for text-to-text generation tasks.

🖙 He finally returned the French book to Chris $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ Austin that I bought last week

B.16 Chinese Antonym and Synonym Substitution

This transformation substitutes Chinese words with their synonyms or antonyms by using the Chinese dictionaryChinese Dictionary: https://github.com/guotong1988/chinese_dictionary and NLP Chinese Data Augmentation dictionaryNLP Chinese Data Augmentation: https://github.com/425776024/nlpcda.

B.17 Chinese Pinyin Butter Fingers Perturbation

This transformation implements the Butter Fingers Perturbation for Chinese characters. Few Chinese words and characters that are picked at random will be substituted with others that have similar pinyin (based on the default Pinyin keyboards in Windows and Mac OS). It uses a database of 16142 Chinese characters https://github.com/pwxcoo/chinese-xinhua and its associated pinyins to generate the perturbations for Chinese characters. A smaller database of 3500 https://github.com/elephantnose/characters more frequently seen Chinese characters are also used in the perturbations with a higher probability of being used compared to less frequently seen Chinese characters. It also uses a database of 575173 words http://thuocl.thunlp.org/ that are combined from several sources https://github.com/fighting41love/Chinese_from_dongxiexidian in order to generate perturbations for Chinese words.

B.18 Chinese Person Named Entities and Gender Perturbation

This perturbation adds noise to all types of text sources containing Chinese names (sentence, paragraph, etc.) by swapping a Chinese name with another Chinese name whilst also allowing the possibility of gender swap. CLUENER Xu et al. (2020); Zhao et al. (2019) is used for tagging named entities in Chinese. The list of names is taken from the Chinese Names Corpus! Yunfei . It can provide assistance in detecting biases present in language models and the ability to infer implicit gender information when presented with gender-specific names. This can also be useful in mitigating representation biases in the input text.

B.19 Chinese (Simplified & Traditional) Perturbation

This perturbation adds noise to all types of text sources containing Chinese words and characters (sentence, paragraph, etc.) by changing the words and characters between Simplified and Traditional Chinese as well as other variants of Chinese Characters such as Japanese Kanji, character-level and phrase-level conversion, character variant conversion and regional idioms among Mainland China, Taiwan and Hong Kong, all available as configurations originally in the OpenChineseConvert project https://github.com/BYVoid/OpenCC.

B.20 City Names Transformation

This transformation replaces instances of populous and well-known cities in Spanish and English sentences with instances of less populous and less well-known cities to help reveal demographic biases Mishra et al. (2020) prevelant in named entity recognition models. The choice of cities have been taken from the World Cities Dataset https://www.kaggle.com/juanmah/world-cities.

🖙 The team was established in Dallas $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ Viera West in 1898 and was a charter member of the NFL in 1920.

B.21 Close Homophones Swap

Humans are generally guided by their senses and are unconsciously robust against phonetic attacks. Such types of attacks are highly popular in languages like English which has an irregular mapping between pronunciation and spelling Eger and Benz (2020). This transformation mimics writing behaviors where users swap words with similar homophones either intentionally or by accident. This transformation acts like a perturbation to test robustness. Few words picked at random are replaced with words with similar homophones which sound similar or look similar. Some of the word choices might not be completely natural to normal human behavior, since humans "prefer" some words over others even they sound exactly the same. So it might not be fully reflecting the natural distribution of intentional or unintentional swapping of words.

🖙 Sentences with gapping, such as Paul likes coffee and Mary tea $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ Tee , lack an overt predicate to indicate the $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ Thee relation between two or more $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ Morr arguments.

B.22 Color Transformation

This transformation augments the input sentence by randomly replacing mentioned colors with different ones from the 147 extended color keywords specified by the World Wide Web Consortium (W3C) https://www.w3.org/TR/2021/REC-css-color-3-20210805/. Some of the colors include “dark sea green”, “misty rose”, “burly wood”.

🖙 Tom bought 3 apples, 1 orange $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ misty rose , and 4 bananas and paid $10.

B.23 Concatenate Two Random Sentences (Bilingual)

Given a dataset, this transformation concatenates a sentence with a previously occuring sentence as explained in Nguyen et al. (2021). A monolingual version is mentioned in the subsequent subsection below. This concatenation would benefit all text tasks that use a transformer (and likely other sequence-to-sequence architectures). Previously published work Nguyen et al. (2021) has shown a large gain in performance of low-resource machine translation using this method. In particular, the learned model is stronger due to being able to see training data that has context diversity, length diversity, and (to a lesser extent) position shifting.

B.24 Concatenate Two Random Sentences (Monolingual)

This is the monolingual counterpart of the above.

🖙 I am just generating a very very very long sentence to make sure that the method is able to handle it. It does not even need to be a sentence. Right? This is not splitting on punctuation... I am just generating a very very very long sentence to make sure that the method is able to handle it. It does not even need to be a sentence. Right? This is not splitting on punctuation...

B.25 Concept2Sentence

This transformation intakes a sentence, its associated integer label, and (optionally) a dataset name that is supported by huggingface/datasets Lhoest et al. (2021a, b). It works by extracting keyword concepts from the original sentence, passing them into a BART Lewis et al. (2020) transformer trained on CommonGen Lin et al. (2019) to generate a new, related sentence which reflects the extracted concepts. Providing a dataset allows the function to use transformers-interpret Pierse (2021) to identify the most critical concepts for use in the generative step. Underneath the hood, this transform makes use of the Sibyl tool Harel-Canada (2021), which is capable of also transforming the label as well. However, this particular implementation of C2S generates new text that is invariant (INV) with respect to the label. Since the model is trained on CommonGen, which is focussed on image captioning, the style of the output sentence would be geared towards scenic descritions and might not necessarily adhere to the syntax of the original sentence. Besides, it can be hard to argue that a handful subset of keywords could provide a complete description of the original sentence.

B.26 Contextual Meaning Perturbation

This transformation was designed to model the "Chinese Whispers" or "Telephone" children’s game: The transformed sentence appears fluent and somewhat logical, but the meaning of the original sentence might not be preserved. To achieve logical coherence, a pre-trained language model is used to replace words with alternatives that match the context of the sentence. Grammar mistakes are reduced by limiting the type of words considered for changes (based on POS tagging) and replacing adjectives with adjectives, nouns with nouns, etc. where possible.

This transformation benefits users who seek perturbations that preserve fluency but not the meaning of the sentence. For instance, it can be used in scenarios where the meaning is relevant to the task, but the model shows a tendency to over-rely on simpler features such as the grammatical correctness and general coherence of the sentence. A real-world example would be the training of quality estimation models for machine translation (does the translation maintain the meaning of the source?) or for text summarisation (does the summary capture the content of the source?).

Word substitution with pre-trained language models has been explored in different settings. For example, the augmentation library nlpaug Ma (2019) and the adversarial attack library TextAttack Morris et al. (2020) include contextual perturbation methods. However, their implementations do not offer control over the type of words that should be perturbed and introduce a large number of grammar mistakes. If the aim is to change the sentence’s meaning while preserving its fluency, this transformation can help to get the same effect with significantly fewer grammatical errors. Li et al. (2020a) propose an alternative approach to achieve a similar objective.

B.27 Contractions and Expansions Perturbation

This perturbation substitutes the text with popular expansions and contractions, e.g., “I’m” is changed to “I am”and vice versa. The list of commonly used contractions & expansions and the implementation of perturbation has been taken from Checklist Ribeiro et al. (2020).

🖙 He often does n’t $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ not come to school.

B.28 Correct Common Misspellings

This transformation acts like a lightweight spell-checker and corrects common misspellings appearing in text by looking for words in Wikipedia’s Lists of Common Misspellings.

🖙 Andrew andd $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ and Alice finally returnd $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ returned the French book that I bought lastr $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ last week

B.29 Country/State Abbreviation

This transformation replaces country and state names with their common abbreviationsCountries States Cities Database: https://github.com/dr5hn/countries-states-cities-database. Abbreviations can be common across different locations: 🖙 “MH” can refer to Country Meath in Ireland as well as the state of Maharashtra in India and hence this transformation might result in a slight loss of information, especially if the surrounding context doesn’t have enough signals.

🖙 One health officer and one epidemiologist have boarded the ship in San Diego, CA $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ California on April 13, 2015 to conduct an environmental health assessment.

B.30 Decontextualisation of the main Event

Semantic Role Labelling (SRL) is a powerful shallow semantic representation to determine who did what to whom, when, and where (and why and how etc). The core arguments generally talk about the participants involved in the event. Addtionally, contextual arguments on the other hand provide more specific information about the event. After tagging a sentence with an appropriate semantic role labels using an SRL labeller Jindal et al. (2020); Shi and Lin (2019a). This transformation crops out contextual arguments to create a new sentence with a minimal description of the event. Helping to generate textual pairs for entailment.

B.31 Diacritic Removal

“Diacritics are marks placed above or below (or sometimes next to) a letter in a word to indicate a particular pronunciation — in regard to accent, tone, or stress — as well as meaning, especially when a homograph exists without the marked letter or letters.” Merriam-Webster . This transformation removes these diacritics or accented characters, and replaces them with their non-accented versions. It can be common for non-native or inexperienced speakers to miss out on any accents and specify non-accented versions.

🖙 She lookèd $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ looked east an she lookèd $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ looked west.

B.32 Disability/Differently Abled Transformation

Disrespectful language can make people feel excluded and represent an obstacle towards their full participation in the society Res (2006). This low-coverage transformation substitutes outdated references to references of disabilities with more appropriate and respectful ones which avoid negative connotations. A small list of inclusive words and phrases have been taken from a public article on inclusive communication, Wikipedia’s list of disability-related terms with negative connotations, terms to avoid while writing about disability.

🖙 They are deaf $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ person or people with a hearing disability.

B.33 Discourse Marker Substitution

This perturbation replaces a discourse marker in a sentence by a semantically equivalent marker. Previous work has identified discourse markers that have low ambiguity Pitler et al. (2008). This transformation uses the corpus analysis on PDTB 2.0 Prasad et al. (2008) to identify discourse markers that are associated with a discourse relation with a chance of at least 0.5. Then, a marker is replaced with a different marker that is associated to the same semantic class.

🖙 It has plunged 13% since $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ inasmuch as July to around 26 cents a pound. A year ago ethylene sold for 33 cents

B.34 Diverse Paraphrase Generation Using SubModular Optimization and Diverse Beam Search

This transformation generates multiple paraphrases of a sentence by employing 4 candidate selection methods on top of a base set of backtranslation models. 1) DiPS Kumar et al. (2019) 2) Diverse Beam Search Vijayakumar et al. (2018) 3) Beam Search Wiseman and Rush (2016) 4) Random. Unlike beam search which generally focusses on the top-k candidates, DiPS introduces a novel formulation of using submodular optimisation to focus on generating more diverse paraphrases and has been proven to be an effective data augmenter for tasks like intent recognition and paraphrase detection Kumar et al. (2019). Diverse Beam Search attempts to generate diverse sequences by employing a diversity promoting alternative to the classical beam search Wiseman and Rush (2016).

B.35 Dislexia Words Swap

This transformation acts like a perturbation by altering some words of the sentences with abberations Board (2021) that are likely to happen in the context of dyslexia.

🖙 Biden hails your $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ you’re relationship with Australia just days after new partnership drew ire from France.

B.36 Emoji Icon Transformation

This transformation converts emojis into their equivalent keyboard format (e.g., -> ":)" ) and vice versa (e.g., ":)" -> ).

B.37 Emojify

This transformation augments the input sentence by swapping words with emojis of similar meanings. Emojis, introduced in 1997 as a set of pictograms used in digital messaging, have become deeply integrated into our daily communication. More than 10% of tweetshttps://blog.twitter.com/en_us/a/2015/emoji-usage-in-tv-conversation and more than 35% of Instagram postshttps://instagram-engineering.com/ include one or more emojis in 2015. Given the ubiquitousness of emojis, there is a growing body of work researching the linguistic and cultural aspects of emojis Guntuku et al. (2019) and how we can leverage the use of emojis to help solve NLP tasks Eisner et al. (2016).

🖙 Apple is looking at buying U.K. startup for $132 billion.$\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$is at startup for$ .

B.38 English Inflectional Variation

This transformation adds inflectional variation to English words and can be used to test the robustness of models against inflectional variations. In English, each inflection generally maps to a Part-Of-Speech tag Penn TreeBank POS in the Penn Treebank Marcus et al. (1993). For each content word in the sentence, it is first lemmatised before randomly sampling a valid POS category and reinflecting the word according to the new category. The sampling process for each word is constrained using its POS tag to maintain the original sense for polysemous words. This has been adapted from the Morpheus Tan et al. (2020) adversarial attack.

🖙 Ujjal Dev Dosanjh served $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ serve as 33rd Premier $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ Premiers of British Columbia from 2000 to 2001

B.39 English Mention Replacement for NER

This transformation randomly swaps an entity mention with another entity mention of the same entity type. Exploiting this transformation as a data augmentation strategy has been empirically shown to improve the performance of underlying (NER) models Dai and Adel (2020).

B.40 Filler Word Augmentation

This augmentation adds noise in the form of colloquial filler phrases. 23 different phrases are chosen across 3 different categories: general filler words and phrases ("uhm", "err", "actually", "like", "you know"…), phrases emphasizing speaker opinion/mental state ("I think/believe/mean", "I would say"…) & phrases indicating uncertainty ("maybe", "perhaps", "probably", "possibly", "most likely").The latter two categories had shown promising results Kovatchev et al. (2021) when they were concatenated at the beginning of the sentence unlike this implementation which perform insertions at any random positions. Filler words are based on the work of Laserna et al. (2014) but have not been explored in the context of data augmentation.

B.41 Style Transfer from Informal to Formal

This transformation transfers the style of text from formal to informal and vice versa. It uses the implementation of Styleformer Damodaran .

🖙 What you upto $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ currently doing ?

B.42 French Conjugation Substitution

This transformation change the conjugation of verbs for simple french sentences with a specified tense. It detects the pronouns used in the sentence in order to conjugate accordingly whenever a sentence contains differents verbs. This version only works for indicative tenses. It also only works for simple direct sentences (subject, verb, COD/COI), which contains a pronoun as subject (il, elle, je etc.). It does not detect when the subject is a couple of nouns ("les enfants" or "la jeune femme").

B.43 Gender And Culture Diversity Name Changer (1-way and 2-way)

Corpora exhibits many representational biases and this transformation focuses on one particular mediator, the personal names. It diversifies names in the corpora along two critical dimensions, gender and cultural background. Technically, the transformation samples a (country, gender) pair and then randomly draws a name from that (country, gender) pair to replace the original name. We collected 42812 distinct names from 141 countries.They are primarily from the World Gender Name Dictionary Raffo (2021).

Common name augmentations do not consider their gender and cultural implication. Thus, they do not necessarily mitigate biases or promote the minority’s representation because the augmented name may be from the same gender and cultural background. This is the case, for example in the CheckList’s Ribeiro et al. (2020) implemented name augmentation. Taking the interaction of the names therein with ours, 34.0%, 33.5%, 31.9%, 30.8% of them are popular names in US, Canada, Australia, and UK, respectively. Only 0.4%, 0.4%, 0.5%, 2.1% of them are from India, Korea, China, and Kazakhstan.

🖙 Rachel $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ Charity Green, a sheltered but friendly woman, flees her wedding day and wealthy yet unfulfilling life.

B.44 Neopronoun Substitution

This transformation performs grammatically correct substitution from English to English of the gendered pronouns, he/she, in a given sentence with their neopronoun counterparts, based on a list compiled by UNC Greensboro and LGBTA WIKIhttps://intercultural.uncg.edu/wp-content/uploads/Neopronouns-Explained-UNCG-Intercultural-Engagement.pdf. NLP models, such as those for neural machine translation, often fail to recognize the neopronouns and treat them as proper nouns. This transformation seeks to render the training data used in NLP pipelines more neopronoun aware to reduce the risk of trans-erasure. The reason why a simple look-up-table approach might not work is due to the fact that the case may differ depending on the context.

🖙 She $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ They had her $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ their friends tell her $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ them about the event.

B.45 Gender Neutral Rewrite

This transformation involves rewriting an English sentence containing a single gendered entity with its gender-neutral variant. One application is machine translation, when translating from a language with gender-neutral pronouns (e.g. Turkish) to a language with gendered pronouns (e.g. English). This transformation is based on the algorithm proposed by Sun et al. (2021).

🖙 His $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ Their dream is to be a fireman $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ firefighter when he $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ they grows $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ grow up.

B.46 GenderSwapper

This transformation introduces gender diversity to the given data. If used as data augmentation for training, the transformation might mitigate gender bias, as shown in Dinan et al. (2020). It also might be used to create a gender-balanced evaluation dataset to expose the gender bias of pre-trained models. This transformation performs lexical substitution of the opposite gender. The list of gender pairs (shepherd <–> shepherdess) is taken from Lu et al. (2019). Genderwise names used from Ribeiro et al. (2020) are also randomly swapped.

B.47 GeoNames Transformation

This transformation augments the input sentence with information based on location entities (specifically cities and countries) available in the GeoNames databasehttp://download.geonames.org/export/dump/. E.g., if a country name is found, the name of the country is appended with information about the country like its capital city, its neighbouring countries, its continent, etc. Some initial ideas of this nature were explored in Pais (2019).

B.48 German Gender Swap

This transformation replaces the masculine nouns and pronouns with their female counterparts for German sentences from a total of 2226 common German names.https://de.wiktionary.org/wiki/Verzeichnis:Deutsch/Namen

🖙 Er $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ Sie ist ein Arzt $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ eine Ärztin und mein Vater $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ meine Mutter .

B.49 Grapheme to Phoneme Substitution

This transformation adds noise to a sentence by randomly converting words to their phonemes. Grapheme-to-phoneme substitution is useful in NLP systems operating on speech. An example of grapheme to phoneme substitution is “permit” P ER0 M IH1 T’.

B.50 Greetings and Farewells

This transformation replaces greetings (e.g. "Hi", "Howdy") and farewells (e.g. "See you", "Good night") with their synonymous equivalents.

🖙 Hey $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ Hi everyone. It’s nice $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ Pleased to meet you. How have $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ are you been ?

B.51 Hashtagify

This transformation modifies an input sentence by identifying named entities and other common words and turning them into hashtags, as often used in social media.

B.52 Insert English and French Abbreviations

This perturbation replaces in texts some well known English and French words or expressions with (one of) their abbreviations. Many of the abbreviations covered here are quite common on social medias platforms, even though some of them are quite generic. This implementation is partly inspired by recent work in Machine Translation Berard et al. (2019).

B.53 Leet Transformation

Visual perturbations are often used to disguise offensive comments on social media (e.g., “!d10t”) or as a distinct writing style (“1337” in “leet speak”) Eger et al. (2019a), especially common in scenarios like video gaming. Humans are unconsciously robust to such visually similar texts. This perturbation replaces letters with their visually similar “leet” counterparts.https://simple.wikipedia.org/wiki/Leet

🖙 Ujjal Dev Dosanjh served $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ U7jal 0ev D0san74 serv3d as 33rd Premier of British Columbia from $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ Pr33i3r 0f 8ritis4 00lu36ia fr0m 2000 to $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ t0 2001

B.54 Lexical Counterfactual Generator

This transformation generates counterfactuals by simply substituting negative words like “not”, “neither” in one sentence of a semantically similar sentence pair. The substituted sentence is then backtranslated in an attempt to correct for grammaticality. This transformation would be useful for tasks like entailment and paraphrase detection.

B.55 Longer Location for NER

This transformation augments data for Named Entity Recognition (NER) tasks by augmenting examples which have a Location Tag. Names of locations are expanded by appending them with cardinal directions like “south”, “N”, “northwest”, etc. The transformation ensures that the tags of the new sentence are accordingly modified.

B.56 Longer Location Names for testing NER

This transformation augments data for Named Entity Recognition (NER) tasks by augmenting examples that have a Location (LOC) Tag. Names of location are expanded by inserting random prefix or postfix word(s). The transformation also ensures that the labels of the new tags are accordingly modified.

B.57 Longer Names for NER

This transformation augments data for Named Entity Recognition (NER) tasks by augmenting examples which have a Person Tag. Names of people are expanded by inserting random characters as initials. The transformation also ensures that the labels of the new tags are accordingly modified.

B.58 Lost in Translation

This transformation is a generalization of the BackTranslation transformation to any sequence of languages supported by the Helsinki-NLP OpusMT models (Tiedemann and Thottingal, 2020).

🖙 Andrew finally returned $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ brought Chris back the French book the French book I bought last week I bought last week

B.59 Mixed Language Perturbation

Mixed language training has been effective for cross-lingual tasks Liu et al. (2020), to help generate data for low-resource scenarios Liu et al. (2021) and for multilingual translation Fan et al. (2021). Two transformations translate randomly picked words in the text from English to other languages (e.g., German). It can be used to test the robustness of a model in a multilingual setting.

🖙 Andrew finally returned the $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ die Comic book to Chris that I bought last week $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ woche

B.60 Mix transliteration

This transformation transliterates randomly picked words from the input sentence (of given source languae script) to a target language script. It can be used to train/test multilingual models to improve/evaluate their ability to understand complete or partially transliterated text.

B.61 MR Value Replacement

This perturbation adds noise to a key-value meaning representation (MR) (and its corresponding sentence) by randomly substituting values/words with their synonyms (or related words). This transformation uses a simple strategy to align values of a MR and tokens in the corresponding sentence inspired by how synonyms are substituted for tasks like machine translation Fadaee et al. (2017). This way, there could be some problems in complex sentences. Besides, the transformation might introduce non-grammatical segments.

B.62 Multilingual Back Translation

This transformation translates a given sentence from a given language into a pivot language and then back to the original language. This transformation is a simple paraphraser that works on 100 different languages. Back Translation has been quite popular now and has been a quick way to augment Li and Specia (2019); Sugiyama and Yoshinaga (2019); Fan et al. (2020).

🖙 Being honest $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ Honesty should be one of our most important character traits $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ characteristics

B.63 Multilingual Dictionary Based Code Switch

This transformation generates multi-lingual code-switching data to fine-tune encoders of large language models Qin et al. (2020); Tan and Joty (2021); Wang et al. (2019b) by making use of bilingual dictionaries of MUSE Lample et al. (2018).

B.64 Multilingual Lexicon Perturbation

This perturbation helps to creates code-mixed sentences for both high-resource and low-resource languages by randomly translating words with a specified probability from any supported languages (e.g., English) to other supported languages (e.g., Chinese) by using a multilingual lexicon. Thus, it can be used to generate code-mixed training data to improve models for multilingual and cross-lingual settings. As of now 100 languages are supported and 3000 common English words listed on ef.com https://www.ef.com/wwen/english-resources/english-vocabulary/top-3000-words/ are supported. The lexicon implementation is also 160x faster than its model based counterpart.

B.65 Causal Negation & Strengthening

This transformation is targeted at augmenting Causal Relations in text and adapts the code from the paper Causal Augmentation for Causal Sentence Classification Tan et al. (2021a). There are two operations: 1. Causal Negation: Negative words like "not, no, did not" are introduced into sentences to unlink the causal relation. 2. Causal Strengthening: Causal meaning is strengthened by converting weaker modal words into stronger ones like "may" to "will" to assert causal strength.

The implementation provides users with the option to amend causal meaning automatically from the root word of the sentence, or by explicitly highlighting the index of the word they wish to amend. Additionally, we include WordNet Miller (1998) synonyms and tense matching to allow for more natural augmentations.

🖙 The rs7044343 polymorphism could be $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$ was involved in regulating the production of IL-33.

B.66 Question Rephrasing transformation

This implementation rephrases questions for sentence tasks by using the T5 model used in B.75 for Question Answering tasks.

B.67 English Noun Compound Paraphraser [N+N]

This transformation replaces two-word noun compounds with a paraphrase, based on the compound paraphrase dataset from SemEval 2013 Task 4 Hendrickx et al. (2013). It currently only works for English. Any two-word compound that appears in a dataset of noun compound paraphrases will be replaced by a paraphrase. If more than one two-word compound appears, then all combinations of compound paraphrases (including no paraphrase at all) will be returned. For example, the paraphrases of “club house” include “house for club activities”, “house for club members”, “house in which a club meets”, etc. We start with replacing paraphrases with the highest score (the specified frequency in the annotated dataset), and paraphrases with the same score (ties) are sorted randomly. This transformation currently only checks for noun compounds from Hendrickx et al. (2013) and therefore has low coverage. To improve it, other datasets could be added, e.g., from Ponkiya et al. (2018) or Lauer (1995). To attain even wider-coverage (at the expense of lower precision), machine learning approaches such as Shwartz and Dagan (2018) or Ponkiya et al. (2020) could be considered. In addition, some of the the paraphrases in Hendrickx et al. (2013) sound a little odd (e.g., "blood cell" -> "cell of blood") and may not fit well in context.

B.68 Number to Word

This transformation acts like a perturbation to improve robustness on processing numerical values. The perturbated sentence contains the same information as the initial sentence but with a different representation of numbers.

B.69 Numeric to Word

This transformation translates numbers in numeric form to their textual representations. This includes general numbers, long numbers, basic math characters, currency, date, time, phone numbers, etc.

B.70 OCR Perturbation

This transformation directly induces Optical Character Recognition (OCR) errors into the input text. It renders the input sentence as an image and recognizes the rendered text using the OCR engine Tesseract 4 (Smith, 2007). It works with text in English, French, Spanish, and German. The implementation follows previous work by Namysl et al. (2021).

B.71 Add Noun Definition

This transformation appends noun definitions onto the original nouns in a sentence. Definitions of nouns are collected from Wikidata https://www.wikidata.org/wiki/Wikidata:Main_Page.

B.72 Pig Latin Cipher

This transformation translates the original text into pig latin. Pig Latin is a well-known deterministic transformation of English words, and can be viewed as a cipher which can be deciphered by a human with relative ease. The resulting sentences are completely unlike examples typically used in language model training. As such, this augmentation change the input into inputs which are difficult for a language model to interpret, while being relatively easy for a human to interpret.

B.73 Pinyin Chinese Character Transcription

This transformation transcribes Chinese characters into their Mandarin pronunciation using the Pinyin romanization scheme. The Character-to-Pinyin converter at the core of this transformation is a neural model by Park and Lee (2020).

B.74 SRL Argument Exchange

This perturbation adds noise to all types of English text sources (sentence, paragraph, etc.) proportional to the number of arguments identified by SRL BERT (Shi and Lin, 2019b). Different rules are applied to deterministically modify the sentence in a meaning-preserving manner. Rules look as follows: if ARGM-LOC and ARGM-TMP both present, exchange them.

Example: [ARG0: Alex] [V: left] [ARG2: for Delhi] [ARGM-COM: with his wife] [ARGM-TMP: at 5 pm] . Alex left for Delhi at 5 pm with his wife.

The transformation relies on propbank annotations (Bonial et al., 2012; Kingsbury and Palmer, 2002; Palmer et al., 2005; Gildea and Palmer, 2002).

B.75 ProtAugment Diverse Paraphrasing

This transformation utilizes the ProtAugment method by Dopierre et al. (2021). The paraphrase generation model is a BART model (Lewis et al., 2020), finetuned on the paraphrase generation task using 3 datasets: Google-PAWS (Zhang et al., 2019b), MSR (Dolan and Brockett, 2005), Quorahttps://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs.

When parpahrasing a sentence, the transformation useu Diverse Beam Search (Vijayakumar et al., 2016) to generate diverse outputs. The diversity penalty term is by default set to 0.5 but can be set to custom values. Additionally, the transformation can use the following generation constraints: (1) A fraction of the words in the input sentence are forbidden in the paraphrase (default 0.7). (2) All bi-grams in the input sentence are forbidden in the paraphrase. This means the paraphrase cannot contain any bi-gram that are in the input sentence. This constraint enforces the paraphrase generation model to change the sentence structure.

B.76 Punctuation

This transformation removes/adds punctuation from an English sentence. This transformation was first introduced by Mille et al. (2021) and used as an example implemention for NL-Augmenter.

B.77 Question-Question Paraphraser for QA

This transformation creates new QA pairs by generating question paraphrases from a T5 model fine-tuned on Quora Question pairs Quora Question Pairs. Generated questions can have a very different surface form from the original question making it a strong paraphrase generator. A T5 model Raffel et al. (2019); Wolf et al. (2020) fine tuned https://huggingface.co/ramsrigouthamg/t5_paraphraser on the Quora Question Pairs dataset was being used to generate question paraphrases. This transformation would benefit Question Answering, Question Generation as well as other tasks which could indirectly benefit eg. for dialog tasks Shrivastava et al. (2021); Dhole (2020).

B.78 Question in CAPS

This transformation upper-cases the context of a question answering example. It also adds upper-cased versions of the original answers to the set of acceptable model responses.

B.79 Random Word Deletion

This transformation randomly removes a word with a given probability $p$ (by default 0.25). The transformation relies on whitespace tokenization and thus only works for English and other languages that are segmented via whitespace. Due to the destructive nature of the transformation, it is likely that the meaning of a sequence may be changed as a result of the change. A similar transformation was suggested by Wei and Zou (2019). Word dropout Goldberg (2017) has been common to help models understand unknown words encountered during evaluation by exposing them to this unknown-word condition during training itself.

B.80 Random Upper-Case Transformation

This perturbation adds noise to all types of text sources (sentence, paragraph, etc.) by randomly adding upper cased letters. With a default probably of 0.1, each character in a sequence is upper-cased. This transformation does not rely on a tokenizer and thus works with all languages that have upper and lower-case letters. One limiation of this transformation is that it will not affect a tokenizer that does lower case for all input. A similar transformation was suggested by Wei and Zou (2019). Further improvement of this transformation exists by potentially relying on extreme value theory Jalalzai et al. (2020).

B.81 Double Context QA

This transformation repeats the context of a question answering example. This should not change the result in any way.

B.82 Replace Abbreviations and Acronyms

This transformation changes abbreviations and acronyms appearing in an English text to their expanded form and respectively, changes expanded abbreviations and acronyms appearing in a text to their shorter form. For example, “send this file asap to human resources” might be changed to “send this file as soon as possible to HR”. The list of abbreviations and acronyms used in this transformation where manually gathered focusing on common abbreviations present in business communications. When abbreviation are context-dependent or highly specific, the induced change may change the meaning of a text, or an abbreviation may not be available in the lookup. The transformation was first introduced by Regina et al. (2020).

B.83 Replace Financial Amounts

This transformation replaces financial amounts throughout a text with the same value in a different currency. The replacement changes the amount, the writing format as well as the currency of the financial amount. For example, the sentence “I owe Fred € 20 and I need € 10 for the bus.” might be changed to “I owe Fred 2 906.37 Yen and I need 1 453.19 Yen for the bus.” The transformation was first introduced by Regina et al. (2020).

B.84 Replace Numerical Values

This transformation looks for numerical values in an English text and replaces it with another random value of the same cardinality. For example, “6.9” may be replaced by “4.2”, or “333” by “789”. The transformation was first introduced by Mille et al. (2021).

B.85 Replace Spelling

This transformation adds noise to all types of English text sources (sentence, paragraph, etc.) using corpora of common spelling errors introduced by Deorowicz and Ciura (2005). Each word with a common misspelling is replaced by the version with mistake with a probability $p$ which by default is set to $0.2$.

B.86 Replace nouns with hyponyms or hypernyms

This transformation replaces common nouns with other related words that are either hyponyms or hypernyms. Hyponyms of a word are more specific in meaning (such as a sub-class of the word), eg: ’spoon’ is a hyponym of ’cutlery’. Hypernyms are related words with a broader meaning (such as a generic category /super-class of the word), eg: ’colour’ is a hypernym of ’red’. Not every word will have a hypernym or hyponym.

B.87 Sampled Sentence Additions

This transformation adds generated sentence to all types of English text sources (sentence, paragraph, etc.) by passing the input text to a GPT-2 model (Radford et al., 2019). By default, GPT-XL is used, together with the prompt “paraphrase:” appended to the original text, after which up to 75 tokens are sampled. Since the additional text is sampled from a model, the model may introduce harmful language or generate text that contradicts the earlier text or changes its meaning. The idea to sample one or more additional sentences was first introduced by Jia and Liang (2017a).

B.88 Sentence Reordering

This perturbation adds noise to all types of text sources (paragraph, document, etc.) by randomly shuffling the order of sentences in the input text Lewis et al. (2020). Sentences are first partially decontextualized by resolving coreference Lee et al. (2018).

This transformation is limited to input text that has more than one sentence. There are still cases where coreference can not be enough for decontextualization. For example, there could be occurences of ellipsis as demonstrated by Gangal et al. (2021) or events could be mentioned in a narrative style which makes it difficult to perform re-ordering or shuffling Kočiskỳ et al. (2018) while keeping the context of the discourse intact.

B.89 Emoji Addition for Sentiment Data

This transformation adds positive emojis and smileys to positive sentiment data and negative emojis to negative sentiment data. For non-labelled data, it adds neutral smileys.

B.90 Shuffle Within Segments

In this transformation, a token sequence, for example BIO-tagged, is split into coherent segments. Thus, each segment corresponds to either a mention or a sequence of out-of-mention tokens. For example, a sentence “She did not complain of headache or any other neurological symptoms .” with tags O O O O O B-problem O B-problem I-problem I-problem I-problem O is split into five segments: [She did not complain of], [headache], [or], [any other neurological symptoms], [.]. Then for each segment, a binomial distribution (p=0.5) is used to decide whether it should be shuffled. If yes, the order of the tokens within the segment is shuffled while the label order is kept unchanged. This transformation is inspired by Dai and Adel (2020).

B.91 Simple Ciphers

This transformation modifies the text in ways that a human could rapidly decipher, but which make the input sequences almost completely unlike typical input sequences which are used during language model training. This transformation includes the following text modifications: double the characters, double the words, add spaces between the characters, reverse all characters in the text, reverse the characters within each word, reverse the order of the words in the text, substitute homoglyphs, rot13 cipher.

B.92 Slangificator

This transformation replaces some of the words (in particular, nouns, adjectives, and adverbs) of an English text with their corresponding slang. The replacement is done with the subset of the "Dictionary of English Slang & Colloquialisms".http://www.peevish.co.uk/slang/index.htm The amount of replacement is proportional to the corresponding probabilities of replacement (by default, 0.5 for nouns, adjectives, and adverbs each).

B.93 Spanish Gender Swap

This transformation changes the gender of all animate entities (mostly referring to people, and some animals) in a given Spanish sentence from masculine to feminine. This includes masculine nouns with feminine equivalents (e.g., doctor → doctora), nouns with a common gender (“sustantivos comunes en cuanto al género”, e.g., el violinista → la violinista), personal pronouns, and (optionally) given names often used with a given gender (e.g., Pedro → Alicia). Epicene nouns are excluded. In addition, the gender of adjectives, determiners, pronouns and participles are modified in order to maintain the grammatical agreement.

B.94 Speech Disfluency Perturbation

This perturbation randomly inserts speech disfluencies in the form of filler words into English texts. With a given probability (0.2 by default), a speech disfluency is inserted between words. The default disfluencies are "um", "uh", "erm", "ah", and "er". At least one filler word is always inserted by this transformation.

B.95 Paraphrasing through Style Transfer

This transformation provides a range of possible styles of writing English language. The following styles can be chosen:

Shakespeare - Trained on written works by Shakespeare.

Switchboard - Trained on a collection of conversational speech transcripts.

Romantic poetry - Trained on romantic poetry.

Basic - A light, basic paraphraser with no specific style.

The transformation follows the models and formulations by Krishna et al. (2020).

B.96 Subject Object Switch

This transformation switches the subject and object of English sentences to generate new sentences with a very high surface similarity but very different meaning. This can be used, for example, for augmenting data for models that assess Semantic Similarity.

B.97 Sentence Summarizaiton

This transformation compresses English sentences by extracting subjects, verbs, and objects of the sentence. It also retains any negations. For example, “Stillwater is not a 2010 American live-action/animated dark fantasy adventure film” turns into “Stillwater !is film”. Zhang et al. (2021) used a similar idea to this transformation.

B.98 Suspecting Paraphraser for QA

This paraphraser transforms a yes/no question into a declarative sentence with a question tag https://www.englishclub.com/grammar/tag-questions.htm, which helps to add more question specific informality to the dataset. Example: ”Did the American National Shipment company really break its own fleet?” -> ”The American National Shipment company really broke its own fleet, didn’t it”.

B.99 Swap Characters Perturbation

This perturbation randomly swaps two adjacent characters in a sentence or a paragraph with a default probability Zhang et al. (2019a).

B.100 Synonym Insertion

This perturbation adds noise to all types of text sources (sentence, paragraph, etc.) by randomly inserting synonyms of randomly selected words excluding punctuations and stopwords Marivate and Sefara (2020).

B.101 Synonym Substitution

This perturbation randomly substitutes some words in an English text with their WordNet Miller (1998) synonyms.

B.102 Syntactically Diverse Paraphrasing using Sow Reap models

This transformation is capable of generating multiple syntactically diverse paraphrases for a given sentence based on the work of Goyal and Durrett (2020). The model paraphrases inputs using a two step framework: 1) SOW (Source Order reWriting): This step enumerates multiple feasible syntactic transformations of the input sentence. 2) REAP (REarrangement Aware Paraphrasing): This step conditions on the multiple reorderings/ rearragements produced by SOW and outputs diverse paraphrases corresponding to these reoderings. The transformation is designed to work only on single-sentence inputs. Multi-sentence inputs results in an empty string/no transformation. The model are trained on the ParaNMT-50M dataset Wieting and Gimpel (2017); Wieting et al. (2017), which can be argued to be a bit noisy.

B.103 Subsequence Substitution for Sequence Tagging

This transformation performs same-label subsequence substitution for the task of sequence tagging, which replaces a subsequence of the input tokens with another one that has the same sequence of tags Shi et al. (2021). This is done as follows: (1) Draw a subsequence A from the input (tokens, tags) tuple. (2) Draw a subsequence B within the whole dataset, with the same tag subsequence. (3) Substitute A with B in the input example.

B.104 Change English Tense

This transformation converts English sentences from one tense to the other, for example simple present to simple past. This transformation was introduced by Logeswaran et al. (2018).

B.105 Token Replacement Based on Lookup Tables

This transformation replaces input tokens with their perturbed versions sampled from a given lookup table of replacement candidates. Lookup tables containing OCR errors and misspellings from prior work are given as examples. Thus, by default, the transformation induces plausible OCR errors and human typos to the input sentence.

The transformation is an adapted and improved version of the lookup table-based noise induction method from Namysl et al. (2020). The OCR lookup table is from Namysl et al. (2021) and the misspellings from Piktus et al. (2019).

B.106 Transformer Fill

This perturbation replaces words based on recommendations from a masked language model. The transformation can limit replacements to certain POS tags (all enabled by default). Many previous papers have used this technique for data augmentation (Ribeiro et al., 2020; Li et al., 2020b, inter alia).

B.107 Underscore Trick

This perturbation adds noise to the text sources like sentence, paragraph, etc. This transformation acts like a perturbation to test robustness. It replaces some random spaces with underscores (or even other selected symbols). This perturbation would benefit all tasks which have a sentence/paragraph/document as input like text classification and text generation, especially on tasks related to understanding/generating scripts.

B.108 Unit converter

This transformation converts length and weight measures to different units (e.g., kilometers to miles) picking at random the new unit but converting accurately the quantity. The transformation conserves the format of the original quantity: "100 pounds" is converted to "1600 ounces" but "one-hundred pounds" is converted to "one thousand, six hundred ounces". Generated transformations display high similarity to the source sentences.

B.109 Urban Thesaurus Swap

This perturbation randomly picks nouns from the input source to convert to related terms drawn from the Urban Dictionary https://www.urbandictionary.com/ resource. It can be applied to an input text to produce semantically-similar output texts in order to generate more robust test sets. We first select nouns at random, then query the Urban Thesaurus website https://urbanthesaurus.org/ to obtain a list of related terms to swap in Wilson et al. (2020).

B.110 Use Acronyms

This transformation changes groups of words for their equivalent acronyms. It’s a simple substitution of groups of words for their acronyms. It helps to increase the size of the dataset as well as improving the understanding of acronyms of models trained on data augmented with this transformation. This transformations works to increase the data for any task that has input texts. It is specially interesting for tasks on semantic similarity, where models should be aware of the equivalence between a set of words and their acronym. The quality of the transformation depends on the list of acronyms. As of now, this list was scraped from wikipedia’s List of Acronyms https://en.wikipedia.org/wiki/Lists_of_acronyms and naively filtered, which leaves space for improvement .

B.111 Visual Attack Letter

This perturbation replaces letters with visually similar, but different, letters. Every letter was embedded into 576-dimensions. The nearest neighbors are obtained through cosine distance. To obtain the embeddings the letter was resized into a 24x24 image, then flattened and scaled. This follows the Image Based Character Embedding (ICES) Eger et al. (2019a).

The top neighbors from each letter are chosen. Some were removed by judgment (e.g. the nearest neighbors for ’v’ are many variations of the letter ’y’) which did not qualify from the image embedding Eger et al. (2019b).

B.112 Weekday Month Abbreviation

This transformation abbreviates or expands the names of months and weekdays, e.g. Mon. -> Monday. Generated transformations display high similarity to the source sentences and does not alter the meaning and the semantic of the original texts. It does not abbreviate plural names, e.g. Sundays, and does not influence texts without names of weekdays or months.

B.113 Whitespace Perturbation

This perturbation adds noise to text by randomly removing or adding whitespaces.

B.114 Context Noise for QA

This transformation chooses a set of words at random from the context and the question and forms a sentence out of them. The sentence is then prepended or appended to the context to create a new QA pair. The transformation is inspired by the the AddAny method described in Adversarial SQUAD Jia and Liang (2017b). However, instead of probing the model to generate adversaries, random words from the context and the question are simply selected and joined together into a sentence, ignoring grammaticality. The transformation attempts to create novel QA pairs assuming that the introduction of random words to the context is less likely to change the answer choice to an asked question.

B.115 Writing System Replacement

This transformation replaces the writing system of the input with another writing system. We use CJK Unified Ideographshttps://en.wikipedia.org/wiki/CJK_Unified_Ideographs as the source of characters for the generated writing systems. The transformation would benefit text classification tasks, especially in the cases where the input writing system is undeciphered.

B.116 Yes-No Question Perturbation

This transformation turns English non-compound statements into yes-no questions. The generated questions can be answered by the statements that were used to generate them. The text is left largely unchanged other than the fronted/modified/added auxiliaries and be-verbs.

The transformation works by getting dependency parse and POS tags from a machine learning model and applying human-engineered, rule-based transformations to those parses/tags. This transformation would particularly benefit question-answering and question-generation tasks, as well as providing surplus legal text for language modeling and masked language modeling.

B.117 Yoda Transformation

This perturbation modifies sentences to flip the clauses such that it reads like "Yoda Speak". For example, "Much to learn, you still have". This form of construction is sometimes called "XSV", where "the “X” being a stand-in for whatever chunk of the sentence goes with the verb", and appears very rarely in English normally. The rarity of this construction in ordinary language makes it particularly well suited for NL augmentation and serves as a relatively easy but potentially powerful test of robustness.

Appendix C Filters

The following is the list of all submitted filters to NL-Augmenter. Filters are used to filter data and create subpopulations of given inputs, according to features such as input complexity, input size, etc. Therefore, the output of a filter is a boolean value, indicating that whether the input meet the filter criterion. We discuss the implementations of each filter alongwith their limitations. The title of each filter subsection is clickable and redirects to the actual python implementation.

This filter identifies whether the input text is code-mixed. It checks that there is at least one sentence in the text where there are tokens representing at least ‘k’ unique languages (with at least a ‘threshold‘ level of confidence that the token is of that language). It is useful for collecting code-mixed data to test the model’s performance on multilingual tasks. The filter relies on ftlidhttps://pypi.org/project/ftlid/ for language detection, therefore, this filter might be limited by the performance of the language detection tool.

🖙 (containing code-mixing) Yo estaba con Esteban yesterday, he was telling me about lo que su esposa vio en los Estados Unidos. $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$✓True

C.2 Diacritics Filter

This filter checks whether any character in the sentence has a diacritic. It can be used to create splits of the dataet where the sentences have diacritics. Accented characters are typically among the rarer characters and checking the model performance on such a split might help investigate model robustness.

🖙 (containing diacritics) She lookèd east an she lookèd west. $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$✓True

C.3 Encoding Filter

This filter filters examples which contain characters outside a given encoding. It can be used to find examples containing e.g. non-ASCII Unicode characters. Filtering out and testing examples that contain these characters can provide feedback on how to improve the models accordingly, since most models are trained with plain English text, which contains mostly ASCII characters. Sometimes non-ASCII character are even explicitly stripped away.

🖙 (containing non-ASCII characters) That souvenir sure was expensive at 60£.. or was it 60€? $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$✓True

C.4 Englishness Filter

This filter identifies texts that contain uniquely British spellings, vocabulary, or slang. The filter uses a vocabulary of common British words/phrases and checks the number of occurence of British words in the given texts. The text is selected if the number exceeds a pre-defined threshold.

🖙 (containing British spellings) Colour is an attribute of light that is perceived by the human eye. $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$✓True

C.5 Gender Bias Filter

This filter filters a text corpus to measure gender fairness with respect to a female gender representation. It supports four languages (i.e. English, French, Polish and Russian) and can be used to define whether the female gender is sufficiently represented in a tested subset of sentences. The filter uses a list of lexicals, which includes filter categories such as personal pronouns, words defining the relation, titles and names, corresponding to the female and male genders accordingly.

🖙 (texts with unbalanced representation) "He went home", "He drives a car", "She has returned" $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$✓True

C.6 Group Inequity Filter

This is a bilingual filter (for English and French languages), which helps to discover potential group inequity issues in the text corpus. It is a topic agnostic filter which accepts user-defined parameters, consisting of keywords inherent to minor group (which potentially might suffer from the discrimination), major group, minor factor and major factor. The filter first flags the sentences as belonging to the minor, and the major groups, and then, the sentences from each of the groups are used to define the intersection with both factors. The filter then compares whether the percentage of major factors exceeds that of the minor factors to determine if the sentences have group inequity issues.

🖙 (containing group inequity issues) "He is a doctor", "She is a nurse", "She works at the hospital" $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$✓True

C.7 Keyword Filter

This is a simple filter, which filters examples based on a pre-defined set of keywords. It can be useful in creating splits for a specific domain.

🖙 (containing keyword "at") Andrew played cricket in India $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$✓True

C.8 Language Filter

This filter selects texts that match any of a given set of ISO 639-1 language codes (the default language being English). Language matching is performed using a pre-trained langid.py model instance. The model provides normalized confidence scores. A minimum threshold score needs to be set, and all sentences with confidence scores above this threshold are accepted by the filter.

🖙 (is English texts) Mein Luftkissenfahrzeug ist voller Aale $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$✗False

C.9 Length Filter

This filter filters data with the input text length matching a specified threshold. It can be useful in creating data with different length distributions.

🖙 (containing more than 3 words) Andrew played cricket in India $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$✓True

C.10 Named-entity-count Filter

This filter filters data where the number of Named Entities in the input match a specified threshold (based on the supported conditions).

🖙 (containing more than 1 named entity) Novak Djokovic is the greatest tennis player of all time. $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$✓True

C.11 Numeric Filter

This filter filters example which contain a numeric value. In the tasks like textual entailment, question answering etc., a quantity (number) could directly affect the final label/response. This filter can be used to create splits to measure the performance separately on texts containing numeric values.

🖙 (containing numbers in texts) John bought a car worth dollar twenty five thousand . $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$✓True

C.12 Oscillatory Hallucinations Filter

This filter is designed to operate in text generation systems’ outputs, with the purpose of extracting oscillatory hallucinations. Oscillatory hallucinations are one class of hallucinations characterized by repeating bigram structure in the outputRaunak et al. (2021). Typically, these behaviors are observed in models trained on noisy corpora. The filter counts the frequency of bigrams in both source and target texts, and compare the frequency difference with a pre-set threshold to determine whether the texts includes oscillatory hallucinations.

🖙 (containing hallucinations in target texts) Source: "Community, European Parliament common regional policy, Mediterranean region", Target: "Arbeitsbedingungen, berufliche Bildung, berufliche Bildung, berufliche Bildung" $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$✓True

C.13 Polarity Filter

This filter filters a transformed text if it does not retain the same polarity as an original text. This filter helps not to distort training data during augmentation for sentiment analysis-related tasks. While generating new data for a sentiment analysis task, it is important to make sure that generated data is labelled correctly.

🖙 (texts retaining polarity) "Hotel is terrible", "Hotel is great" $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$✗False

C.14 Quantitative Question Filter

This is a simple rule-based filter that can be used to identify quantitative questions. It can help to analyse models’ performance on questions which require numerical understanding. It is also useful to study possible biases in question generation.

🖙 (being quantitative question) How long does the journey take? $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$✓True

C.15 Question type filter

This filter helps identify the question category of a question answering example based on the question word or the named entity type of the answer. Knowledge of the question type can help in the development of question answering systems Parikh et al. (2019) as well as for assessing performance on individual splits.

🖙 (being where question) Where is Delhi located ? $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$✓True

C.16 Repetitions Filter

This filter finds texts with repetitions with simple heuristic rules. It might be helpful in finding repetitions that frequently occur in the spoken language data.

🖙 (containing repetitions in texts) I I want to sleep $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$✓True

C.17 Phonetic Match Filter

This filter selects texts that contain matching entries to a list of supplied keywords. It first transform the input sentence and the keywords into phenetic units and then compare whether the two phenetic unit sets have overlap.

🖙 (containing homophones of keyword "trombone") I left my trombno on the train $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$✓True

C.18 Special Casing Filter

This filter checks if the input sentence has a special casing, i.e. the string is either all lowercased, all uppercased or has title casing. It might be useful for creating splits that contain texts with unusual casing, e.g. misspellings.

🖙 (text being uppercased/lowercased/titlecased) let’s go to chipotle $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$✓True

C.19 Speech-Tag Filter

This filter filters an example text based on a set of speech tags and identifies whether the count of selected POS tags meet the pre-defined conditions (e.g. above the threshold).

🖙 (containing 1 verb and 2 numbers in texts) It all happened between November 2007 and November 2008. $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$✓True

C.20 Token-Amount filter

This filter filters an example text based on whether certain keywords are present in a specified amount.

🖙 (containing 2 occurances of "in") Andrew played cricket in a soccer stadium in India at 9pm $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$✓True

C.21 Toxicity Filter

This filter filters an example text which has a toxicity value matching a particular threshold. It uses a pre-trained toxicity detector, which can provide 7 toxicity scores. All the 7 types of toxicity scores can be used as criteria for the filtering.

🖙 (text being toxic) I disagree. It is not supposed to work that way. $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$✗False

C.22 Universal Bias Filter

This filter works the same way as the Gender Bias Filter, but measures balance of representation for more categories (religion, race, ethnicity, gender, sexual orientation, age, appearance, disability, experience, education, economic status). The lexical seeds representing these categories are currently available in English only, however the pool of languages can be extended by a simple addition of the lexical seeds in a desired language to the lexicals.json file.

🖙 (texts being biased) "He is going to make a cake.", "She is going to program", "Nobody likes washing dishes", "She agreed to help him" $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$✗False

C.23 Yes/no question filter

This filter allows to select questions that can be correctly answered with either ’yes’ or ’no’. Since it is rule-based, the limitation of this filter is that questions that are ambiguous might not be recognized.

🖙 (text being yes/no question) Wasn’t she angry when you told her about the accident? $\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}$✓True