I improved my model based off of Kerpathy’s makeMore part two tutorial and a 2003 paper by bengio et al (link is https://jmlr.org/papers/volume3/tmp/bengio03a.pdf), it was really interesting. I increased the input size, so it takes in the previous 3 letters as an input instead of 1, and encoded each letter in a two dimensions instead of one hot encoding. I also had to start using numPy due to needing to do efficient matrix multiplication (numPy does it in C in a heavily optimized fashion, I considered trying to do this but decided it was out of the scope of the project). Doing it in just python would be way too slow.
I also optimized it a little bit, having it select 32 examples randomly from the data set for each epoch, instead of just using the whole thing, which made it a lot faster while still training over the whole set. Hopefully this will make it produce more accurate results!