@28taidan

Fixed up the readme a bit in preparation to ship. Specifically, I added info on what all the files do, but I named them poorly and then forgot what most of them do so it’s kind of confusing.

Trained the model a little more, exported it, wrote some UI to make it look more like a chatbot. It still sounds like gibberish. Sometimes I imagine that it’s output changes slightly depending on the context (for example, more numbers if I ask it a number related question, and a list if I ask it for a list, etc), but I’m probably just imagining it, it’s really just gibberish. Still kinda cool tho. You can check out the demo at https://luchik28.github.io/BadGPT/.

Added an EOS ID, which is a special token that the model can output to indicate that it’s done. I built a script to insert this in between passages in the training data, so it can learn when it should shut up (otherwise it will yap forever, until it hits the max number of output tokens). This meant I had to retrain the entire model, but got it down to a loss of around 3.46. Obviously, it’s not very good. But I’m pretty sure the only way to improve is to scale up the model size and training time, something I don’t really have the capacity to do (this model trained for 15 hours on my laptop). I will probably continue by adding this model to my demo website, and build a chatGPT like UI, just for fun. After that, I think it might be time to ship!

Switched to using the fineweb edu dataset, instead of shakespear. This is the dataset I will use to train my actual model on. It’s a highly currated database of the internet, and is similar to what GPT2 was trained on (although the actual db they trained it on is closed source). I also reformated the sampling system so you can put in a query and get a response, as long as the query takes up less tokens than the context. Other than swapping out the training file, it’s the exact same. Next, I will probably implement a special ending token to my tokenizer, and also increase the scale of my model a little bit. I can’t increase it all the way to what Karpathy does in his tutorial (he uses cloud computing, so he paid to rent out like 8 64GB GPUs, or something like that), since I don’t want to spend money training it. However I think I can squeeze a little more out of my laptop.

Started working on a more advanced tokenizer. The tokenizer is what splits the input text from words into numbers, and back again. Before, I was using a 10 dimensional embedding, where each letter is represented by a 10 long array (each value representing a dimension). But current models have much more advanced tokenizers, which often encompass entire words, or parts of words for longer words. This works using Byte Pair Encoding, which turns all of the letters pairs of letters into their utf encodings (usually 8), then iteratively merging the most common consecutive pairs into tokens to add to the vocabulary. You can see how tokenizers work at https://tiktokenizer.vercel.app/, which is a cool website that lets you play around with stuff like that.

Updated the demo website to include the wavenet model, and the new self-attention shakespear model. You can check it out here.

I finished training it! It finished with a val loss of around 1.76, it got down to 1.73 but ticked up at the end. The output is pretty good, but not substantially better than the wavenet model. However, it’s vaguely word-like, and took 20 hours to train so I’m not sure how much larger I can make the model before my laptop implodes. Next I will probably try to find a question-answer training dataset, so that I can start training it to answer questions. This step that I’ve been working on is the pre-training step, where I teach the model to basically spit out documents. Next, I have to have it spit out documents that happen to be question-answer documents.

I made a bunch of tiny improvements. I didn’t change the architecture at all, but I did change the optimizer to AdamW. The optimizer is what controls how much each parameter is changed each epoch. Before, I was using a set learning rate, of about .1 for the first 80% of training then decreasing to .01 for the last 20%. Decreasing it at the end is important because as the loss settles into a local minimum, it might not be able to go to the lowest point because it keeps overshooting it since the lr is too high. Conversely, we can’t keep the lr too small the whole time because it might settle into a very tiny local minima that it might have been able to go over with a higher learning rate. It would also take forever. AdamW helps improve the optimizing step by changing the learning rate per parameter, helping the model train as fast as possible without overfitting. Overfitting isn’t really an issue I’m worrying about since my model is pretty small (around 56k params), so it’s not really large enough to learn the ins and outs of my dataset. Anyway I also changed the logging step, so it now logs validation dataset loss in addition to the training dataset loss. I split the dataset into train and validation so I can make sure it’s not overfitting. Currently the val loss is at 1.76, which is a lot better than where we were at before, which makes me hopeful. Training has around another 7,000 epochs to go, but Stardance is yelling at me I should write another devlog now so here you go, Stardance. I’m looking forward to sampling the model and seeing how much it’s improved!

I implemented self attention! I will try to explain here, but will do a terrible job: basically, what that is, is it lets the model decide what tokens in the input are more important than the others. The first thing I did was scrap the wavenet model, and switched to a simple bigram model. The second thing I implemented was bag of words, which edits the input tokens so every element is the average of that element and all that came before it. This lets all of the tokens “talk” to eachother. This is all fine and dandy but since all the things are averaged then you don’t know where things came from, and positional data is lost, so we lose a lot of resolution and data, so that sucks. So we have to encode positional data. So instead of just averaging all the elements that came before, we expand each averaged element into a list, which has all the elements in a row instead of averaging them into one number. So the first row could be [1, 0, 0] and the second [1, 2, 0] and the third [1, 2, 3], etc. We do this by masking it with the tril function, from numpy (very cool). But this is problematic, because all of the elements are given the exact same weight. A letter 34 characters back gets the same weight as the last character in the input (which is the letter immediately preceding the predicted letter, the output). To fix this, we add self attention! This is modeled after the groundbreaking paper Attention is All You Need, which started the whole AI hype thing. How it works is each token has a linear layer (that multiplies it by learned weights) called the key, and another called the query. The query is kind of like what that token is looking for, and the key is kind of like what that token has. And these can be multiplied with the keys and queries of all the other tokens, and that lets you know which token has the stuff that the most other tokens are looking for, and is thus more important and can be given more weight. That is a single head of attention, there are multiple heads that can specialize in different things. This, in theory, should increase performance. But it also drastically increases the amount of time it takes to train. The model only has ~56k parameters, which is less than half of the wavenet model, but takes a lot more time to do it’s thing. This is the output from the results of more than 13 hours of training. As you can see from the loss function, the loss was decreasing at a steady rate, so more training could probably improve it. Plus, this was at a highish learning rate the whole time (.1) so decreasing it would probably further improve it. But I can’t do anything on my computer while this is training or it runs out of RAM, and I have to use my computer now, so I will be moving on for now.

I increased the block size from 8 characters to 64. This means that instead of the model taking in the previous 8 characters and outputting the next one, it takes in the previous 64 characters and outputs the next. This means that it should be able to keep track of longer words so it doesn’t get lost in them. However, it doesn’t seem to be improving the model too much, although that might be because I haven’t trained it as much as the others. I’ve done roughly 200,000 epochs, which took a long time, since this model is much larger (110,415 parameters vs 45,000), and has 6 hidden layers as apposed to 2, which means it is a lot slower. As you can see in the loss graph, it probably could make further gains, although it seems to be mostly good.

Trained the wavenet architecture on a file with just shakespear in it. It’s got the structure down well, and sometimes can do words and wordlike outputs, but generally spouts straight gibberish. I will try to follow a lecture to build a better Tokenizer and Transformer next time. Right now each character is treated as a token, and embedded into 10 dimensions, it would be better if pairs of characters were treated as tokens and embedded. Increasing the context size could also help, it’s currently an input of the previous 8 characters and outputting 1. This is it’s output:

Implemented the wavenet architecture. Usually, during the flattening phase, the tensor goes from the shape (batch num, block size, embedding dimensions) to (batch num, block size * embedding dimensions). This squishes all of the embedded characters into one dimension. Wavenet does that more gradually, it slowly merges the different letters. So the first two letter embeddings get squished into one, then the next two, etc, and in the next layer then you have the first 4, then the next four, until by the final layer you have just one group of all the letters, which is the same as the original method, just a much more gradual approach. The first image is the training loss, the huge cliff is when the learning rate decreased. The second image is a visualization of the wavenet architecture from the paper. You can see that in each layer, nearby groups of input get merged together. In the original method, the graph would look like all of the inputs going to the 1st dot in the hidden layer, then continuing up. Next I will be actually building a GPT instead of this names project.

I did a bunch of things to prep for implementing a wavenet architecture, which basically changes the model so instead of taking in all the characters of input at once and squishing them all into the first layer, it gradually puts them in at each separate layer. First, I put more stuff in classes! This didn’t really change the functionality, but instead of embedding the characters and flattening the model in separate chunks, I put them in their own classes so they can get put in the list of layers, and we can just run the input through each layer in the list, and compare it with the target at the end, instead of having half the layers in the list and half as separate chunks, which is a lot smoother. So the entire model is just:

model = Sequential([  
    Embedding(vocab_size, n_embd),
    FlattenConsecutive(2), Linear(n_embd * 2, n_hidden, bias=False), BatchNorm1D(n_hidden), Tanh(),  
    FlattenConsecutive(2), Linear(n_hidden * 2, n_hidden, bias=False), BatchNorm1D(n_hidden), Tanh(),  
    FlattenConsecutive(2), Linear(n_hidden * 2, n_hidden, bias=False), BatchNorm1D(n_hidden), Tanh(),  
    Linear(n_hidden, vocab_size),
])

Which I think is really cool. I also improved the loss graph, previously it looked like the screenshot on the right, because each iteration’s loss is graphed. Because our batch size is 32, which means in each iteration we look at just 32 inputs/output, a good deal of what the loss will be depends on how lucky we are with the input/output. Over a great number of iterations, it averages out into the correct value, though, so to fix the graph, we instead plot the average of every 1000 iterations, which lets us see a lot more detail that otherwise would be lost in the noise (the two loss graphs I added are not of the same model, but you get the idea). Anyway now I’m going to actually implement the wavenet architecture!

I followed Karpathy’s lecture on backpropagation, where he implemented backpropagation manually. In the series, he’s using Pytorch, which does it for you, which is why he’s doing it manually. I’m not using Pytorch, so the only way to do it is to do it manually, so I didn’t get to add too much to the code, I just verified that I had implemented it correctly and learned a bunch. It is still pretty confusing though, and I couldn’t solve any of the exercises in the lecture without watching his explanations. Anyway here’s a name it generated that I think is pretty cool.

Added a demo page! You can check it out here: https://luchik28.github.io/BadGPT/. I saved the model weights, built a script that reconstructs the model using those weights, and hosted it using github pages.

Put my model into the Layer class, so I can easily create lots of layers. The graph shows the activation distribution by layer. Each layer is pretty similar, as you can see. This devlog marks the end of the fourth lecture in Andrej Karpathy’s makemore series, out of 7, so we’re getting there! That’s just under 8 hours or lecture watched (so I spend roughly 5 minutes coding for every minute of lecture watched). Pretty interesting.

Wrote all of the initial code, am currently testing it on livestream data from the 2026 Kalahari tournament (roughly 24 hours of video, and 350 matches). It goes very slowly, it’s been running for like 5 hours and still has a good chunk to go. The screenshot is of the visualization software with a demo match loaded.

I was playing around with how changing the number of epochs/other variable affected the output, and seeing how that differed to before I implemented Batch normalization, and so tried to put the epochs up a lot. I ended up training it for 20,000,000 epochs, which took like 8 hours, which is why this devlog has so much time logged. It didn’t really improve it much, you can see by the loss graph it didn’t go down very much at all, so I’ll need to change something in how the code works. It was still fun though.

I worked on switching to a Jupyter notebook instead of just a python file, this made it easier since I didn’t have to constantly retrain the model, and also could print out stuff in line. I also added batch normalization, which helps change the range (really the standard deviation) of the values before going into the activation function (tanh) so that the values are more spread out. tanh squishes all values between -1 and 1, so if all of the numbers are greater than 1 or less than -1, then all of the values will essentially be squished to either -1 or 1, which makes the model lose a lot of the data. But if you divide all the numbers so that they are all closer to 0, then the activation function is a lot more spread out and the data is preserved. (I got the photo from google, I think it’s helpful when visualizing it).