Recurrent Neural Network

After training it with the updated code, the output was terrible although the error was much lower, so I was really confused. I wasn’t sure what went wrong, and it took me a very long time to figure it out. When training, I feed in the previous step’s memory back into it and in training the memory is never wiped each epoch. However, when testing the output, I initialised it with a memory of just zeros, which it had barely experienced training with so wasn’t able to output meaningful things. I was able to fix it by initialising its memory on the training text itself to give it a memory state it is used to and that fixed it. You can see an output from it below.

I did multiple training sessions and tweaked the learning rate, as after training with more layers and complexity, the model ended up with a much worse rate. I tried decreasing the learning rate and that led to a much better result. I then implemented something which reduces the learning rate over time, so that hopefully it will not plateau. I am thinking that to get to more levels of complexity in terms of producing actual content i will need to turn it into a LSTM, which has long and short memory. I am going to train it for a while now with the update and see how it goes.

I updated the rnn to support multiple hidden layers, as previously it had a single layer, so I had to update the forward and backwards passes as well as the adagrad to account for the multiple layers. I started off with two and tested it with a hidden size of 256, but I didn’t see any improvement, however I didn’t train it for as long. I am now in the process of implementing a variable number of layers from 1 to any number, so that the model can learn greater complexity, however this comes at the cost of much slower training. Currently the training on 2 layers is a decent slowdown compared to with 1 layer, as i also increased hidden size to 512. So far I have made the forward pass work for multiple layers (greater than 2) but the rest of it still only supports 2 layers.

I am making a recurrent neural network from scratch in python using numpy, trained on Shakespeare text to generate character-by-character in his style.This is an improvement on the previous neural network that I made as it keeps a memory based on the things it has recently seen, so it’s next prediction depends on what it has previously seen which will hopefully make it be able to learn more advanced patterns.I added loading and chunking of data, so basically it takes in a text file, and then it splits it into chunks (I use 25 character long chunks) in pairs with one shifted across by one and saves it as input and targets.I wrote the forward pass, loops through each character in a chunk, updates a hidden state with tanh, then a softmax over the vocab to predict the next character.I implemented backprop through time, which was definitely the hardest part, had a few off-by-one bugs in the hidden state indexing that took a while to track down.I did a numerical gradient check to double-check the backward pass against finite differences before training it, and realised I forgot to update the bias gradient so it wasn’t training properly.I used adagrad as the matrix updater from the gradients which works by updating the gradients by a smaller and smaller amount each time (I looked it up and apparently it is a good way to do it).I wrote the output function where I give it an input function, and then the model predicts the next character, and then it feeds the output back into the model and repeats for as many characters as I want.I then trained it on Shakespeare text and ran 100k iterations at hidden size 100, took about 2 mins and started outputting vaguely word-shaped gibberish with correct capitalisation and character-name formatting.I then increased the hidden size from 100 to 256 and trained it more, and it still looks like gibberish but improved a bit, some of the output is what you can see in the screenshot.