Backpropagation

It’s learning! I tried the network with a small synthetic dataset and it learned the pattern flawlessly, at least when you don’t let it run for too long:

Below you can see the network’s performance (cost) at each epoch, it starts out strong - reducing the cost and getting better and better - until all of a sudden there is a massive spike and the cost goes way up again.

I have no idea why this is happening, but overall I’m extremely happy that the network is learning.

Next I’m going to fix this issue and then I want it to predict numbers from the MNIST dataset.