After training it with the updated code, the output was terrible although the error was much lower, so I was really confused. I wasn’t sure what went wrong, and it took me a very long time to figure it out. When training, I feed in the previous step’s memory back into it and in training the memory is never wiped each epoch. However, when testing the output, I initialised it with a memory of just zeros, which it had barely experienced training with so wasn’t able to output meaningful things. I was able to fix it by initialising its memory on the training text itself to give it a memory state it is used to and that fixed it. You can see an output from it below.