Java Neural Network from Scratch

These past few days, I have been working on almost entirely training/testing and fine-tuning.

• I added the ability to change activation function per neuron, which can be used in rare cases
• I mainly added a webcam ability, which is shown in the pictures below.
  It’s not very accurate, which can be a bad selection of network configurations or a bad dataset, which I’m leaning towards since the majority of it was just fruits on a white background with a few pictures I took myself.
  It also means the example weights and biases or not very accurate for fruits, as seen in the 3rd picture. It keeps thinking random objects are lime and tracking the wrong things, but with the right settings and a good dataset, I’m sure it can be good.
• The predictions are also happening in 3 ms, which is really good.

Over these past 3 days I was able to add a better prediction and training for my convolutional neural network.

• im2col: I switched from traditional sliding over the image, which was inefficient, to an algorithm that enhances it. It first extracts patches by sliding the filter across the input tensor, extracting every patch (they overlap too). Each patch is 3D, so it must be flattened into a vector and then stacked to form a 2D array. The weights are then also stacked to form a 2D array. Finally, the 2 matrices are multiplied.
• backpropagation: The CNN also needs to learn, so it accumulates gradients, which it uses to calculate how much the error changes by modifying weights and bias. The opposite of im2col is also used to calculate errors. However, in the beginning, the dense layers also need to pass their errors back, so I made a method that passes errors, specifically for CNN backpropagation.
• Pooling layers: pooling layers are very important for CNNs, as they reduce the size of the feature maps, which can help cut down computing costs. It can also help the model recognize an object, even if its position is shifted slightly.
• I might implement a GPU-accelerated CNN using tornadoVM, but this will be challenging, as I may have to modify methods. However, for larger-scale networks, this might add a large performance boost.

Over the past 2 days when I did not post a devlog. I managed to get a lot done.
Instead of relying on hiddenLayersDepth and hiddenLayersWidth, the neteork now uses an array of layer sizes, allowing for more flexibility.
I also added training based on Dataset classes, which are much easier for data organization by using ArrayLists (It didnt change the network architecture so I still am using arrays).
I needed to convert 2D weight/bias arrays into ArrayLists, which made a static method in the util class for.
After that, I did some cleanup by removing redundant variables, which made the code much less messy.
Today, I worked on allowing to customize the activation functions used. For now you can only change one function for the whole hidden layer section, but I will add the ability to make an array later. I used an Enum and a bunch of methods from the util class to add relu, leaky relu, tanh, sigmoid, and softmax to the Activation enum.
Also today, I started work on my convolutional neural network, which is the architecture used for object recognition. I finished most of the layers necessary, but I am yet to test its prediction for errors, so I have not pushed it to github yet. I have the filters and the feed forward network already done, so it shouldn’t be that hard.

I added saving and loading the neural network - The network saves to a .bin file with a desired path, saving space. In a previous project I made a while ago, I used JSON files, which grew very large. This will hopefully save space.
I made a new method in my util class. This helps convert a 1D array into a 2D one with a key, telling how many elements per layer. This will help when I switch over from hiddenLayersWidth and hiddenLayersDepth to an array of lengths.

I added a better training method - Mini-Batch Gradient Descent. It trains the network in small batches instead of the whole dataset. I spent a few hours fixing bugs from implementing this.
Attatched are screenshot of the network’s training performance. Although it is a small network being trained on a simple task, it is still nice to see that the network is running well.
After that, I decided to try to implement training from the GPU using TornadoVM. It did not work, so I did not push to my repository. Once I get it working, however, I will.

Neural network from scratch - added basic training

I added a way to train the model, calculating deltas and adjusting for the errors. Here, I trained it to output the XOR values based on the inputs, where -1 is false and 1 is true. The network outputted answers which appeared to be true (very close to -1 and 1), and before they were outputting random noise. This shows that the network learned something.

Neural network has a predict() method

Although it currently outputs a random value since it can’t be trained yet, it is still a good milestone.

I tested a network with 3 input neurons, 4 output neurons, and 2 hidden layers containing 4 neurons each. I ran the predictions 1 billion times based on random inputs. Here are the results:

Total time: 167380.87 ms
Speed per prediction: 0.000167 ms
Per second: 5,974,399 predictions per second

Neural network - neuron compute function added and changed double to float for better performance