It actually learns

After getting XOR working, I tried my neural net on MNIST (handwritten digit recognition). Loading the data was easier than expected, though getting it into the right shape for my network took some fiddling. To make sure I loaded it correctly, I drew a few of the digits to check they looked right before training.

Once that was sorted, training worked well, and the results were better than I hoped: 97.8% accuracy, in about 5 minutes.

I also experimented with running the whole thing in a browser, but that turned out to be more involved than expected, so I’m parking it for now and focusing on other improvements first.

It learns (a bit)

After implementing loss functions, activation functions, and the forward and backward pass for the layers, I tried training a simple network to predict the XOR function.

After running the network, I found that it didn’t learn at all. The loss just stayed at 0.25 without any changes. Not only that, the whole neural network didn’t update anything.

After some initial digging, I had a feeling this was one of those dumb, simple bugs you can’t find for hours. Unfortunately, my feeling was right: the error was multiplying the bias instead of adding it. This is really bad because the bias initially has a value of 0, which means all outputs turn out to be 0 (not good).

Of course, the fix was a one character change: * to +.

Now that the network can predict XOR, I’ll see if it can also handle MNIST. I’m still a bit scared about performance issues with matrix multiplication.

Rust Learning Project: Building a Neural Network

I recently read a few chapters of The Rust Programming Language and wanted to put my newly gained knowledge to use. Naturally, the first thing that came to mind was building a neural network in Rust. I’d already tried this in Python and failed, so I figured maybe, for some reason, it would go better this time around.

Matmul

The first step was implementing matrices and their operations in Rust, the most important of course being matrix multiplication. Since this would likely become the biggest bottleneck for the whole model, I got curious about its performance and benchmarked multiplying two 1024x1024 matrices.

The naive implementation took about 20 seconds. One simple optimization brought that down to 14 seconds.

Then I built with compiler optimizations enabled, and the naive implementation suddenly ran in 63 nanoseconds. I was floored, briefly convinced I’d stumbled onto some incredible compiler magic, before realizing what actually happened: the compiler had simply skipped the calculation entirely, since the result was never used. Classic dead code elimination.

After forcing the compiler to keep the computation, the naive implementation came in at a more believable 2 seconds, and the optimized version dropped to 200ms.

For comparison, Numpy does the same multiplication in about 4ms. Whether 200ms is fast enough for training neural nets in reasonable time remains to be seen.