I have made a couple of changes to the embedding of this network. Firstly, I have increased the embedding dimension from 16 to 128, allowing it to capture much more information about the image. I have added a couple of new files to my Github repo, allowing me to view the corrupted images, run the model already trained and have started work on a classifier based on my Siamese Network.
Additionally, I realised what was causing my model to be terrible at predicting two similar images: I had accidentally weighted it to focus on different images at a 10x higher rate than similar images. With these out of the way, I have been able to lower the error rate of my model drastically.
Another thing I have done is change how the corruption function works. Previously, it would normalise the RGB colours to 0-1 by simply dividing by a factor. This caused the images to get darker, lowering my model’s performance. By switching this to a sigmoidal function, I was able to achieve much better results.
I also changed the different img target to rely on the labels. If the two images had the same label but were different, they would be related with a dot product of about 0.9, drastically improving my model’s organizational abilities. By which I mean that the embeddings for similar concepts are themselves similar. Seems obvious, but I forgot to do it until this point.
%20dot%20matrices.png)
Comments 2
This is really cool. Two directions you could take it from here:
Freeze the encoder and train a decoder on top. Since the encoder is already corruption-invariant, this is a clean way to see what the embedding actually retained. Feed it the noisy image, target the clean one, only the decoder gets gradients.
Or train a full VAE end to end for denoised reconstruction. Same idea, but the encoder trains too and you add the reparameterization plus KL term. Loss is computed against the pre-noise image. It’s a pretty fun project.
Also not that the Siamese objective explicitly trains the encoder to throw away corruption info, and it’s compressing 3072 dims down to 256. Expect blurry class level reconstructions rather than sharp ones.
Shoot me a message on discord ‘xerneas3318’ if you have any questions i’d be happy to help
@xerneas Hey, thank you so much for this comment. Currently I am working on straightening out a few kinks with the base model(I tried to improve one thing and everything broke), but your second idea seems really cool! I have already done a little bit on the first one, just need to improve the base model to up performance, but the denoted reconstruction sounds interesting. I will most likely work on a reconstruction quite soon. Thanks again for responding with such a lengthy comment, this is some very cool stuff.
Sign in to join the conversation.