Neural network devlog #3

I finally have a working PCNN!

What was done

After I defined the neural network, I proceeded to train it to predict XOR. However, it failed.. quite miserably.
So, I tried asking ChatGPT what was wrong with my network. But it turns out CGPT was as lost as I was, which delayed the network development.

How I fixed the bug.

After the previous attempt not working, I tried asking ChatGPT to take my existing code, and rewright it as a working example. That worked. Then, I asked another ChatGPT in a new chat to compare my existing (not working) example with the functioning example. This time, it correctly found out what’s wrong.

What was wrong.

Turns out, the theoretic structure, training, and maths behind the network was fully correct. Instead, I made many smaller mistakes that caused a simple problem seem extremely hard for the network. For example:

• Per one example, I gave the network 1000 iterations. Turns out, 50 was enough. The network therefore learned almost 20x slower.
• My network was HUGE. 5 layers, meaning 1 input layer, 1 output layer, and 3 internal layers. 1 internal layer was enough. The network didn’t quite know what to do with the rest.
• In XOR, you have input 0/1 and output 0/1. Neural networks absolutely HATE zeros. Its better to make input -1/+1 and output -1/+1.
• My learning rate was set to 0.01 to make the network ‘stable’ and not learn too quickly. After later tests, turns out it destabilizes at lr=15 and lr=1 is the best for it. At that time, CGPT recommended 0.1 which worked pretty well too.
• I used ReLU as my activation function. Tanh works way better.
• I didn’t batch the training examples.

These small mistakes combined made the network untrainable.

Process

After this, I manually fixed my network (no AI used).

Result

The network can now learn XOR in… basically 50 milliseconds.

Thanks for reading!

Neural network devlog #2

I have implemented a basic PCN(layers) class.
A model consisting of 3 layers might be defined as PCN([128, 64, 10]).

Process

I didn’t train it to do anything yet, but it seems to be working.
I started with learning the basics of numpy, ie., how to make an array, array types, how classes are used etc..
Once I did that, I created a brief .md file of how I would implement the neural network concretely, with all the equations and structure and stuff. Then I went to a LLM (cgpt & gemini) and asked it whether it would work if I implemented it.
Took multiple iterations to get it concretely-right, but eventually it was done.
I started with defining the class. Then, I made the __init__ function, settle, train and reinitialize functions. Again, I asked LLMs for any mistakes, giving it the paper to base it’s answers on. There were a few mistakes in the process which I then fixed.
I used only python+numpy, the whole thing is in one file.
I’d also like to thank Artem Kirsanov (YT) for his amazing video explanation

Result

I think I have the definition of the model done. I need to create the training loop next to test it.

Concept

The idea behind predictive coding is this:
- Each layer has a state, and the layers are connected with matrices.
- Each layer’s state must be capable of successfully predicting the leftwards layer’s state after going through the matrix connecting it to the leftwards layer.
Now, imagine we have a greyscale image consisting of 128 pixels that shows a number 0-9. We can encode it into the leftwards layer.
The rightwards layer must predict the middle one which must predict the leftwards one. However, the image has size 128 whereas the righwards layer has size 10. Therefore, the network is forced to compress the 128-long image into a 10-long vector.
How do we teach it to do this? We randomly initialize the states of the layers. Next, we clamp the left layer to some image. Then, we let the network find a configuration of states that matches the leftwards image best. Of course, it can’t find such a configuration because the matrices aren’t trained yet. Once it has settled, we gently adjust the weight matrices to match the image better.
If we repeat this process enough times, the network finds a configuration for the matrices that let it encode the image into the rightwards layer.

That’s it! Next, I’ll try training the network on a XOR function to see if it’s actually capable of doing anything, maybe even move on to MNIST.
I should also create a github page for this, as I’m currently using gitg locally for this.

Neural Network Devlog #1

I might’ve gotten something wrong, sorry if that happened.
I’m trying to create an implementation of a predictive coding neural network.
I have started with reading the tutorial above, it’s a great source. Written down my own understanding of it, made a couple iterations with ChatGPT to correct any mistakes.

After this I’ll continue with learning the basics of numpy and writing an implementation of it in python.