Neural network devlog #2

I have implemented a basic PCN(layers) class.
A model consisting of 3 layers might be defined as PCN([128, 64, 10]).

Process

I didn’t train it to do anything yet, but it seems to be working.
I started with learning the basics of numpy, ie., how to make an array, array types, how classes are used etc..
Once I did that, I created a brief .md file of how I would implement the neural network concretely, with all the equations and structure and stuff. Then I went to a LLM (cgpt & gemini) and asked it whether it would work if I implemented it.
Took multiple iterations to get it concretely-right, but eventually it was done.
I started with defining the class. Then, I made the __init__ function, settle, train and reinitialize functions. Again, I asked LLMs for any mistakes, giving it the paper to base it’s answers on. There were a few mistakes in the process which I then fixed.
I used only python+numpy, the whole thing is in one file.
I’d also like to thank Artem Kirsanov (YT) for his amazing video explanation

Result

I think I have the definition of the model done. I need to create the training loop next to test it.

Concept

The idea behind predictive coding is this:
- Each layer has a state, and the layers are connected with matrices.
- Each layer’s state must be capable of successfully predicting the leftwards layer’s state after going through the matrix connecting it to the leftwards layer.
Now, imagine we have a greyscale image consisting of 128 pixels that shows a number 0-9. We can encode it into the leftwards layer.
The rightwards layer must predict the middle one which must predict the leftwards one. However, the image has size 128 whereas the righwards layer has size 10. Therefore, the network is forced to compress the 128-long image into a 10-long vector.
How do we teach it to do this? We randomly initialize the states of the layers. Next, we clamp the left layer to some image. Then, we let the network find a configuration of states that matches the leftwards image best. Of course, it can’t find such a configuration because the matrices aren’t trained yet. Once it has settled, we gently adjust the weight matrices to match the image better.
If we repeat this process enough times, the network finds a configuration for the matrices that let it encode the image into the rightwards layer.

That’s it! Next, I’ll try training the network on a XOR function to see if it’s actually capable of doing anything, maybe even move on to MNIST.
I should also create a github page for this, as I’m currently using gitg locally for this.