Recurrent Neural Network

Added some code so that users can pass in arguments. I also spent a while trying to set up the releases page using github actions, and I used claude to make the github actions script for me (none of this went towards logged hours as it wasn’t in the IDE), but I finally got the releases to work (ignore the 1.0.4, I was trying to get it to work). I had three problems. One of them was that I accidentally named the “github” folder “github” (without the dot) so it wasn’t correctly running as an action. Another problem was pyinstaller using relative paths but remembered the same thing happened to me on my last project so i just used the same fix. The final problem was that my model checkpoint file somehow got deleted and it was needed for the Github actions but I had another copy so I put it back into the project and committed it, and it finally worked. I also got Al to help me write the README as it is boring to do. It will be shipped shortly so please check it out. (also dont mind the screenshot below being a screenshot of stardance, when I tried to post this, the site froze, so I screenshotted it first to make sure I didn’t lose it, and then it broke so I did it again).

After training it with the updated code, the output was terrible although the error was much lower, so I was really confused. I wasn’t sure what went wrong, and it took me a very long time to figure it out. When training, I feed in the previous step’s memory back into it and in training the memory is never wiped each epoch. However, when testing the output, I initialised it with a memory of just zeros, which it had barely experienced training with so wasn’t able to output meaningful things. I was able to fix it by initialising its memory on the training text itself to give it a memory state it is used to and that fixed it. You can see an output from it below.

I did multiple training sessions and tweaked the learning rate, as after training with more layers and complexity, the model ended up with a much worse rate. I tried decreasing the learning rate and that led to a much better result. I then implemented something which reduces the learning rate over time, so that hopefully it will not plateau. I am thinking that to get to more levels of complexity in terms of producing actual content i will need to turn it into a LSTM, which has long and short memory. I am going to train it for a while now with the update and see how it goes.

I updated the rnn to support multiple hidden layers, as previously it had a single layer, so I had to update the forward and backwards passes as well as the adagrad to account for the multiple layers. I started off with two and tested it with a hidden size of 256, but I didn’t see any improvement, however I didn’t train it for as long. I am now in the process of implementing a variable number of layers from 1 to any number, so that the model can learn greater complexity, however this comes at the cost of much slower training. Currently the training on 2 layers is a decent slowdown compared to with 1 layer, as i also increased hidden size to 512. So far I have made the forward pass work for multiple layers (greater than 2) but the rest of it still only supports 2 layers.

I am making a recurrent neural network from scratch in python using numpy, trained on Shakespeare text to generate character-by-character in his style.This is an improvement on the previous neural network that I made as it keeps a memory based on the things it has recently seen, so it’s next prediction depends on what it has previously seen which will hopefully make it be able to learn more advanced patterns.I added loading and chunking of data, so basically it takes in a text file, and then it splits it into chunks (I use 25 character long chunks) in pairs with one shifted across by one and saves it as input and targets.I wrote the forward pass, loops through each character in a chunk, updates a hidden state with tanh, then a softmax over the vocab to predict the next character.I implemented backprop through time, which was definitely the hardest part, had a few off-by-one bugs in the hidden state indexing that took a while to track down.I did a numerical gradient check to double-check the backward pass against finite differences before training it, and realised I forgot to update the bias gradient so it wasn’t training properly.I used adagrad as the matrix updater from the gradients which works by updating the gradients by a smaller and smaller amount each time (I looked it up and apparently it is a good way to do it).I wrote the output function where I give it an input function, and then the model predicts the next character, and then it feeds the output back into the model and repeats for as many characters as I want.I then trained it on Shakespeare text and ran 100k iterations at hidden size 100, took about 2 mins and started outputting vaguely word-shaped gibberish with correct capitalisation and character-name formatting.I then increased the hidden size from 100 to 256 and trained it more, and it still looks like gibberish but improved a bit, some of the output is what you can see in the screenshot.