I have added an optimization function to my Siamese Network!

For some information on these images, the difference error graph(the one on the right) is measuring the difference between the high-dimensional representations, or embeddings of different images. I want this graph to be as high as possible, meaning each image gets its own unique embedding. The graph on the left, however, measures the difference in embeddings between corrupted and uncorrupted versions of the *same * image. This should be as low as possible, meaning corrupted and uncorrupted versions of an image are converted into very similar embeddings.

Unfortunately, it is struggling to make the embedding of different images distinct, shown through a very low difference. This may seem bad, but it is working EXACTLY AS EXPECTED.

In many papers on these types of neural networks, cosine similarity is used to reduce feature collapse. I will get into cosine similarity in a later devlog, but for now, I will explain feature collapse. Feature collapse is a phenomenon in neural networks in which all outputs of a network collapse to a single value. This causes the network to always spew the same output value, regardless of the input. In my network, this would show up as a very low similarity error and a very low difference.

This is exactly what is shown in the image below. Therefore, I am on the right track!