Implemented the wavenet architecture. Usually, during the flattening phase, the tensor goes from the shape (batch num, block size, embedding dimensions) to (batch num, block size * embedding dimensions). This squishes all of the embedded characters into one dimension. Wavenet does that more gradually, it slowly merges the different letters. So the first two letter embeddings get squished into one, then the next two, etc, and in the next layer then you have the first 4, then the next four, until by the final layer you have just one group of all the letters, which is the same as the original method, just a much more gradual approach. The first image is the training loss, the huge cliff is when the learning rate decreased. The second image is a visualization of the wavenet architecture from the paper. You can see that in each layer, nearby groups of input get merged together. In the original method, the graph would look like all of the inputs going to the 1st dot in the hidden layer, then continuing up. Next I will be actually building a GPT instead of this names project.