Open comments for this post
Devlog 2 — Training on MNIST
Today I finally got NeuroLab training on real data. I loaded the MNIST dataset, which has 10,000 handwritten digit images, and split it into 8,000 for training and 2,000 for testing.
I wrote a load_mnist function that reads the CSV file, splits the labels from the pixels, and normalizes the pixel values by dividing by 255.0 so they are between 0 and 1. I also wrote a one_hot function that converts digit labels like 5 into arrays like [0,0,0,0,0,1,0,0,0,0] so the network can compare its 10 outputs to the correct answer.
The first training run was a disaster. The loss started at over 1000, which means the network was completely wrong. Turns out the weight initialization was the problem. Switched to He initialization by multiplying the random weights by np.sqrt(2.0 / input_size), and the starting loss dropped all the way to 0.14.
After 100 training cycles, the loss got down to 0.03!
Open comments for this post
Today I started building NeuroLab, a neural network completely from scratch in Python using only NumPy. No TensorFlow, no PyTorch, just pure math and matrices. This is also my CS50P final project and my main Stardance project for the whole summer!
What I built today: I started with the core building blocks. First was sigmoid, which squashes any number between 0 and 1. Then ReLU, which returns 0 for negative numbers and the input itself for positive numbers. I also wrote relu_derivative for backpropagation, init_weights using random NumPy arrays, and mse_loss, which measures how wrong the network is by averaging squared differences.
Then I built the actual NeuralNetwork class with init to set up weights and biases for each layer, forward_pass to run data through the network, forward_cache, which does the same thing but saves intermediate values so backprop can use them, and backpropagation, which adjusts the weights based on how wrong the prediction was.
Problems I ran into: relu_derivative was crashing on arrays, so I fixed it by using .astype on NumPy boolean arrays. My forward_cache loop was also zipping the wrong lists. The worst bug was the gradient direction being completely flipped; the network was doing gradient ascent instead of descent, so the loss kept getting bigger instead of smaller. Fixed it by switching the subtraction order in the loss function. Gradients were also exploding with large batches, so I divided all gradients by m, which is the number of samples. Here is some of the code I wrote (It might be hard to see):
