Word Embeddings
- 7 Devlogs
- 28 Total hours
Word Embeddings transform english words into vectors, so you can calculate with words such as: dog - bark + meow = cat They are also used in search engines, recommendation systems and spam detection.
Ship #1
Pending review
Embedding Demo Website (Pages 1-3)
I really like the way the embeddings turned out. That’s why I want to make a website to make word embeddings as accessible and understandable as possible.
I’m assuming no prior knowledge from the visitors, and try to interactively guide them through the world of embeddings, with each slide explaining something about embeddings or being interactive to try it yourself. This time i finished slides 1-3, with the last one being the interactive one.
Since it should feel engaging and alive, I focus a lot on subtle animations and micro-interactions. I’ve never built a website with heavy animations before so this is the perfect opportunity to try GSAP!
Maths with English words
With the new dataset, training took a lot longer. My computer worked for around 50 hours to run through 26GB of training data.
The long wait really payed off though, the model is so much better than the previous model and it is finally capable of doing the embedding arithmetic I’ve been chasing the whole time.
Embedding arithmetic is so fun, here are some examples:
-
throw - throwing + running -> run(-ing form) -
better - good + friendly -> friendlier(comparasion) -
woman - man + king -> queen(man compared to woman is like king compared to queen) -
germany - berlin + paris(capitals)
The model learned a lot more syntactic and semantic meaning than I had expected.
Next I want to build a web app, so that you can play with the embeddings yourself.
New Dataset + Improved Pre-Processing
In an effort to increase the model’s performance I switched from the previous model’s Wikipedia Simple English dataset to a subset of the Fine Web Edu dataset. As an additional measure to improve the results I decided to use a larger context of 4 words left and right to the root word.
I also completely re-wrote all the preprocessing steps. Even though the dataset is high-quality, language is very nuanced and needs filtering. But since words next to each other (4 words before and after the root word) get grouped, filtering an entire word means the group mustn’t be formed (as information is now missing), on the other hand removing punctuation from the end of a word should still allow that word to appear in a group.
Overall I filter/handle:
- quotes
- brackets
- non-English Characters
- apostrophes
- other special characters (dashes, underscores, etc.)
Another challenge is ensuring that grouped words actually belong together. In the previous dataset I simply grouped the words next to each other in a giant blob of text. This time I only group words from the same text source (the same article, document, …) and only within the sentences of that source. This ensures groups don’t span multiple sentences.
This taught me a lot about working with larger amounts of data - I couldn’t load everything in memory as it would be too large, but had to read from disk in batches to process the data. After ~50 min of leaving my computer untouched I had the final dataset with ~26GB (~18GB compressed using word IDs) of filtered word groups.
Next, I will have to find a way to determine unrelated words, so that I can start training the third iteration of the model.
New (worse) model
In an effort to improve on the previous model, I created a new one. Now it uses separate weights for the context instead of re-using the embedding weights in the context. The new model has 30.000.000 parameters which is double the previous one.
Yet unfortunately, the results I got were disappointingly miserable: In contrast to the previous model, it doesn’t even group related words. According to the embeddings, the word “cat” is very similar in meaning to “motor” or “caution” 😂😭
I don’t know why it’s performing so poorly, but I’m curious to find out.
First Embedding Model
Using the data obtained previously I could get my first embeddings 🎉
The model (image) has 15.000.000 parameters and took roughly 1/2 hour to train. Unfortunately, the performance is below my expectations: It’s not yet good enough so that “uncle - aunt + mum” would output “dad”.
However, the embeddings definitely captured some meaning: For example, the vector of “android” is close to “ios” and to “smartphones”. The vector of “cat” is close to the words “goat”, “monkeys” and “snakes”.
I want to improve this model significantly until it’s good enough to meaningfully calculate with the embedding vectors.
Data preprocessing
Before the model can train to retrieve the embeddings, it needs data. I found a dataset on Kaggle with the text of 249.396 wikipedia articles.
Today I did some preprocessing, first I removed any non-english words and delete special characters,
then I determined the 50.000 most frequent words, and limited my vocabulary to these words.
To compute the embeddings, I took the left and right words of each word in the data and put them into a pandas data frame along with two other unrelated words: (left word, center word, right word, unrelated word 1, unrelated word 2)
The final dataset has 24.780.670 rows, so I likely cant train on the full dataset on my computer.
The goal is for the model, when presented with the center word to predict which of the other words are similar to it which it can only do by learning the meaning of each word - which is what we want.
