@Clark

Heyyy!

I have been making some changes to my Neural Compression Algorithm, resulting in a greatly improved model. These changes fall in 3 major ones:

1. Architecture shift:
   Previously, I had many Convolutional layers that shrunk the spatial dimensions because they had no padding. It seems that adding padding to prevent Convs from shrinking spatial dimensions, and using a pool on top of that is far better.
   ○ I reduced the number of conv layers,
   ○ added more pooling layers,
   ○ added padding, and
   ○ Implemented upscaling.
   This upscaling is really important and possibly the main reason for my model’s newfound performance. Previously, I was relying on linear layers to increase the spatial dimensions, but linear layers are not that good for images. Thus, it was best for me to implement the opposite of a pooling layer, an upsampling layer. Basically, this just splits each pixel into 4 new pixels, where each of these 4 pixels have the same colour as the original. From here, I can use a convolutional layer to smooth out the data.

2. Compression size increase:
   The second major improvement came from increasing the size of the compressed image. However, it may seem wrong to improve the model simply by reducing the work it must do, but I argue this is necessary. Before, I was shrinking an image from 32 * 32 * 3 (32 pixels wide, 32 pixels tall, and 3 for RGB) to just 64 numbers. This results 48x fewer data to store, which is simply too much. If these images were higher quality (like if they were 4K resolution), it would be easier to shrink the images by a similar factor. However, I do not have such images, but that may come in a future iteration

3. Rewrite:
   I have rewritten the entire training file from scratch, removing any unnecessary lines as I go. This cut down on bloat and reduced training time by ~2-3x. A lower training time allowed me to run the training loop for more epochs, greatly speeding up the rate at which I can improve my code.

If you took the time to read this, thank you so much! Feel free to comment and ask me any questions about this project.

Hey there!

I have been working on a neural algorithm to compress images. It compresses a 32 x 32 pixel image down to just 64 numbers. These 64 numbers act as a choke point that the image must pass through. Next, it brings these numbers back up to the 32 x 32 pixel image. By training the model to output exactly the same as its input, this chokepoint can be thought of as a compressed version of the image.

After doing all of this, the model was able to output a very blurry version of the original image. While this may seem bad, it is actually a success! This just means that the image has to pass through a much too small choke point. Unfortunately, this idea is wrong, and I do not yet know exactly what is wrong.

These past 10 hours of coding have been absolute hell. I have tried to make images with the same label closer in the embedding space, but then everything else broke. After about a week, I just gave up on it entirely and accepted that it was only going to aim for differentiating images, not caring about the image labels. After making this change(a very saddening one, pouring all that time down the drain), I was able to spike model performance greatly.
I have practically come full circle, with my model hardly different from what it was at my last devlog, but I have learnt some valuable stuff. I have improved the loss function, giving a much cleaner way to compare embedding vectors and calculate their (dis)similarity. Additionally, I have created a few pieces of code that allow me to visualise various aspects of my model. All in all, I wouldn’t consider this time to be a total waste, as I have learned some valuable stuff. If you want to take a look at the various parts I have added, check out the images attached below and read my explanation of what they do:

1. The first image shows all the kernels in the first convolutional layer of my network.
2. The second image shows a tree with all the operations applied to the images, converting them from an image to an embedding vector.
3. The third image shows the process of my model being trained

I have made a couple of changes to the embedding of this network. Firstly, I have increased the embedding dimension from 16 to 128, allowing it to capture much more information about the image. I have added a couple of new files to my Github repo, allowing me to view the corrupted images, run the model already trained and have started work on a classifier based on my Siamese Network.

Additionally, I realised what was causing my model to be terrible at predicting two similar images: I had accidentally weighted it to focus on different images at a 10x higher rate than similar images. With these out of the way, I have been able to lower the error rate of my model drastically.
Another thing I have done is change how the corruption function works. Previously, it would normalise the RGB colours to 0-1 by simply dividing by a factor. This caused the images to get darker, lowering my model’s performance. By switching this to a sigmoidal function, I was able to achieve much better results.
I also changed the different img target to rely on the labels. If the two images had the same label but were different, they would be related with a dot product of about 0.9, drastically improving my model’s organizational abilities. By which I mean that the embeddings for similar concepts are themselves similar. Seems obvious, but I forgot to do it until this point.

I have been working on an improved optimization algorithm for my Siamese network. I have done this using a dot matrix. What this means is that I have taken the pairwise dot product between the embeddings of two images. These two images are either the corrupted and non-corrupted versions of the same image, or two different images. The two different images will be referred to as Set A, and the two same images will be referred to as Set B.

Set A, the cosine similarity matrix(same image) in the image below, shows very promising results, matching up with what is hoped. A majority of the image is random noise, representing a low correlation between different parts of the embedding vector. This is intended because I want to maximize entropy, meaning it can store the most information in each part of the embedding vector. The main diagonal of Set A(the one going from the top left to the bottom right) shows a very high value. This means that the embedding vectors of the corrupted and non-corrupted versions of an image are very similar. Set B is very noisy, meaning that the embedding vectors of unrelated images are very dissimilar, exactly what I want.

These results are very promising, allowing me to work on interesting uses of this model, which will be the focus of my next few devlogs.

I have added an optimization function to my Siamese Network!

For some information on these images, the difference error graph(the one on the right) is measuring the difference between the high-dimensional representations, or embeddings of different images. I want this graph to be as high as possible, meaning each image gets its own unique embedding. The graph on the left, however, measures the difference in embeddings between corrupted and uncorrupted versions of the *same * image. This should be as low as possible, meaning corrupted and uncorrupted versions of an image are converted into very similar embeddings.

Unfortunately, it is struggling to make the embedding of different images distinct, shown through a very low difference. This may seem bad, but it is working EXACTLY AS EXPECTED.

In many papers on these types of neural networks, cosine similarity is used to reduce feature collapse. I will get into cosine similarity in a later devlog, but for now, I will explain feature collapse. Feature collapse is a phenomenon in neural networks in which all outputs of a network collapse to a single value. This causes the network to always spew the same output value, regardless of the input. In my network, this would show up as a very low similarity error and a very low difference.

This is exactly what is shown in the image below. Therefore, I am on the right track!

Hey there! I have been working on a Siamese network, and this is my first devlog. A Siamese network is a type of neural network akin to an advanced compression algorithm. It works by converting an image into an abstract list of numbers, known as an embedding. It teaches the network to make these abstract numbers as close as possible for corrupted and uncorrupted versions of the same image, and as different as possible for different images. At the moment, I have just been working on the corruption algorithm, which I have got working. Siamese Networks were some of the earliest neural networks applied to real problems, that of recognising handwritten digits for an ATM.

Just a test devlog