Started working on a more advanced tokenizer. The tokenizer is what splits the input text from words into numbers, and back again. Before, I was using a 10 dimensional embedding, where each letter is represented by a 10 long array (each value representing a dimension). But current models have much more advanced tokenizers, which often encompass entire words, or parts of words for longer words. This works using Byte Pair Encoding, which turns all of the letters pairs of letters into their utf encodings (usually 8), then iteratively merging the most common consecutive pairs into tokens to add to the vocabulary. You can see how tokenizers work at https://tiktokenizer.vercel.app/, which is a cool website that lets you play around with stuff like that.