It learns (a bit)

After implementing loss functions, activation functions, and the forward and backward pass for the layers, I tried training a simple network to predict the XOR function.

After running the network, I found that it didn’t learn at all. The loss just stayed at 0.25 without any changes. Not only that, the whole neural network didn’t update anything.

After some initial digging, I had a feeling this was one of those dumb, simple bugs you can’t find for hours. Unfortunately, my feeling was right: the error was multiplying the bias instead of adding it. This is really bad because the bias initially has a value of 0, which means all outputs turn out to be 0 (not good).

Of course, the fix was a one character change: * to +.

Now that the network can predict XOR, I’ll see if it can also handle MNIST. I’m still a bit scared about performance issues with matrix multiplication.