Data preprocessing

Before the model can train to retrieve the embeddings, it needs data. I found a dataset on Kaggle with the text of 249.396 wikipedia articles.

Today I did some preprocessing, first I removed any non-english words and delete special characters,
then I determined the 50.000 most frequent words, and limited my vocabulary to these words.
To compute the embeddings, I took the left and right words of each word in the data and put them into a pandas data frame along with two other unrelated words: (left word, center word, right word, unrelated word 1, unrelated word 2)

The final dataset has 24.780.670 rows, so I likely cant train on the full dataset on my computer.

The goal is for the model, when presented with the center word to predict which of the other words are similar to it which it can only do by learning the meaning of each word - which is what we want.