Who is Who - Self-Fish devlog #3

There were so many duplicate images

So in the last devlog, I told my scraper scraped nearly 150k images, but when I began cleaning them I noticed a lot of images were exact duplicates, I thought it was just for a class or two but It was for every single one of them , except Oreo

After manually cleaning 14 classes with duplicates, I gave up, and decided to use python to delete duplicates, I used hashlib and md5 hash to detect exact duplicate images inside all folders of my root train images folder which each represent one class, and I used os.remove to delete those duplicates.

I ended up deleting around 34k images which brings my dataset from 150k to 100k images, but this would change too cuz the 100k maybe unique but they still have a lot of discrepencies, like random products and all.

I also used matplotlib to plot a graph of the images remaining, and the results dont look good for a few classes which have now less than 300 images, others are in range 500 - 800, remember they were around 1100 images earlier, so around 800 images were duplicates of the remaining 300.

I know what’s causing this, multiple search queries, since google images only displays a fixed 400 images per search, I made my scraper in a way that it searches for the same product multiple times but with a different query which causes popular images to be duplicates

One more thing, the dataset would not only decrease in number, but after cleaning the images, I would augment them which would drastically increase the number of images and also make the model less likely to overfit.

Also the graph is really ugly cuz there are like 130 classes on the x axis and they dont really fit well and I am not an expert in pyplot so I dont know how to make it look good