BERT

Model

Bidirectional Encoder Representations from Transformers or BERT is a pre-trained language model that was developed by Google in 2018. BERT is trained on large amounts of data and is designed to learn the underlying relationships between words in a sentence. BERT has been shown to outperform many existing models on these tasks and has become a popular choice for many NLP applications.

The model is designed as a deep neural network with multiple layers of “transformer” blocks. Each transformer block contains a self-attention mechanism, which allows the model to focus on different parts of the input sentence at different times. The self-attention mechanism also permits the model to learn long-range dependencies between words, which is useful for accurately modeling the meaning of a sentence or long text.

Implementation Introduction

This work implements BERT through the Hugging Face’s Transformers library. which provides pre-trained models to perform multiple NLP tasks on texts. A pre-trained model is one that has been previously trained on a considerably large dataset and saved for prediction or fine-tuning.

The experiments are performed training BERT base model (uncased), which is pretrained model on English language using a masked language modeling (MLM) objective. BERT multilingual base model (uncased), a pretrained model on the top 102 languages with the largest Wikipedia using a masked language modeling (MLM) objective and robertuito-emotion-analysis which is a model trained with TASS 2020 Task 2 corpus for Emotion detection in Spanish. Base model is RoBERTuito, a RoBERTa model trained in Spanish tweets. RoBERTuito emotional analysis is also known as pysentimiento.

Implementation

The experiments were performed on a google collab notebook given the high-efficiency needed to perform this task. Colab notebook