New Dataset + Improved Pre-Processing

In an effort to increase the model’s performance I switched from the previous model’s Wikipedia Simple English dataset to a subset of the Fine Web Edu dataset. As an additional measure to improve the results I decided to use a larger context of 4 words left and right to the root word.

I also completely re-wrote all the preprocessing steps. Even though the dataset is high-quality, language is very nuanced and needs filtering. But since words next to each other (4 words before and after the root word) get grouped, filtering an entire word means the group mustn’t be formed (as information is now missing), on the other hand removing punctuation from the end of a word should still allow that word to appear in a group.
Overall I filter/handle:

• quotes
• brackets
• non-English Characters
• apostrophes
• other special characters (dashes, underscores, etc.)

Another challenge is ensuring that grouped words actually belong together. In the previous dataset I simply grouped the words next to each other in a giant blob of text. This time I only group words from the same text source (the same article, document, …) and only within the sentences of that source. This ensures groups don’t span multiple sentences.

This taught me a lot about working with larger amounts of data - I couldn’t load everything in memory as it would be too large, but had to read from disk in batches to process the data. After ~50 min of leaving my computer untouched I had the final dataset with ~26GB (~18GB compressed using word IDs) of filtered word groups.

Next, I will have to find a way to determine unrelated words, so that I can start training the third iteration of the model.