@william_a_stella

I’ve been working on a BPE tokenizer to make my own LLM from scratch. Basically what this does is it splits the original input text for the LLM into “tokens” that the LLM does computations on and trains on for future steps. It works by creating an initial vocabulary of characters (256 using UTF-8), and then adds to the vocabulary by finding common pairs of characters and adding the most common pair into its vocabulary as a new independent token. I made the initial vocabulary and did “pre-tokenization” which is a method that allows you to compute the most common pairs of characters fast.

I did some work in determining the type and strength of transformations of KV cache layers relative to each other.