GPU Layers

I initially thought porting the forward and backward passes to the GPU would be as simple as replacing Matrix with GpuMatrix everywhere. It wasn’t.

Unlike CPU operations, GPU operations can’t allocate memory themselves. On the CPU, x + 2 simply creates a new value. On the GPU, every operation needs a pre-existing buffer to write its result into. So x += 2 works, but y = x + 2 requires a buffer for y to exist first. And x = x + 2 isn’t supported on the GPU at all.

The solution is to pre-allocate buffers and keep them around, resizing them only when the batch size changes. It works, but it also means every struct starts accumulating buffer fields and scaling that approach to an entire network only makes the problem worse.

Next: some kind of global buffer manager, before tackling the loss function.