I added a better training method - Mini-Batch Gradient Descent. It trains the network in small batches instead of the whole dataset. I spent a few hours fixing bugs from implementing this.
Attatched are screenshot of the network’s training performance. Although it is a small network being trained on a simple task, it is still nice to see that the network is running well.
After that, I decided to try to implement training from the GPU using TornadoVM. It did not work, so I did not push to my repository. Once I get it working, however, I will.