Using Language Embeddings with Synthetic Minority Over-Sampling Technique

• In Event: Tackling Biases in Tricky Data Collection

Abstract

Classification of imbalanced datasets is a well-known problem in machine learning literature (Fernandez et al. 2018). One solution to this problem is SMOTE, the Synthetic Oversampling Minority Technique. SMOTE is an over-sampling approach where the minority class is over-sampled by creating synthetic samples rather than over-sampling with replacement (Chawla et al. 2002). Synthetic samples are generated as follows: (1) Given a data point, a second data point is selected amongst its k-nearest neighbors, and then (2) a random point between the two real samples is selected and used as a new synthetic sample. This approach can reduce potential overfitting while increasing precision, accuracy, and the f1-score of a given classifier.

While SMOTE is widely used in the classification of numeric data, it has failed to increase classification performance when it comes to text-as-data due to large, sparse feature sets when using traditional methods such as bag of words or term frequency-inverse document frequency calculation to generate numeric features. Taking advantage of recent progress in natural language processing, and language embeddings in particular, we extend SMOTE to text data using state-of-the-art language embeddings. Our proposed method, LESMOTE (Language Embeddings for Synthetic Minority Oversampling Technique), combines state of the art language embedding methods (including RoBERTa, BERT, and other transformer-based models) with SMOTE to generate synthetic samples to improve classification for imbalanced text data. We find that LESMOTE improves accuracy and f1-scores for classification tasks across four separate case studies. In addition, we find that adding synthetic minority samples improves classification performance for the majority class(es) without requiring synthetic majority samples.

To generate synthetic text using LESMOTE, we embed the text using a transformer-based model, generate the logits (an output of transformer-based models) for the vocabulary set using the embeddings, and apply SMOTE to the logit space. We are then able to convert the logit embeddings back into text to use as synthetic samples in classification. To test this method, we apply LESMOTE to four datasets with imbalanced classes: (1) tweets labeled with 21 policy labels as defined by the Comparative Agendas Project; (2) tweets about Common Core State Standards labeled as positive, negative, or neutral; (3) tweets about Black Lives Matter labeled as positive, negative, or neutral; and (4) tweets labeled as either political or nonpolitical. We compare results where the synthetic data is included in the training data vs. when the synthetic data is excluded from the training data and find that on out-of-sample prediction (where no synthetic data is included in the out-of-sample prediction task) the f1-score is improved by an average of 27%. LESMOTE is flexible, and can be used in a variety of text classification contexts where training data is imbalanced.

In our paper, we outline the method, present the four case studies, and provide guidance to researchers for applying LESMOTE to their imbalanced text data classification tasks. In addition, we provide an open-source software package for researchers to implement this method.

Authors

• Zhanna Terechshenko, New York University

• Megan Brown, New York University

• Richard Bonneau, NYU

• Joshua A. Tucker, New York University

• Jonathan Nagler, New York University