Self-Fish Devlog #2

My Scraper is on steroids !!!

I scraped 150,000 images

Yooooo guys the scraper worked too good now, it scraped roughly 150,000 images in just 6 hours, that is insane, the speed was around 7 images per second or 420 images per minute, that is really crazy, my old scraper took a full day for 75k images, this is like a million times faster!!!

In the last devlog, I acheived a speed of around 1079 images in 10 minutes or around 100 images per hour, so since then I have made it 4x times faster.

Here’s what I did:

• Added premature scrolling - earlier the scraper would miss a lot of images or would wait for a long period of time for the images to load and then scroll, but now as soon as the main div appears on the screen, the scraper scrolls down a bit triggering lazy load instantly and then scrolls down to the bottom triggering te lazy load 2 times in a cycle, this made the whole scrolling thing so much quicker.

• Fixed downloads not cancelling - I am storing some extra urls so that if one or more of the urls failed to download, I have backup for completing the target, but I am using a ThreadPoolExecutor to download images concurrently, when it reaches the target the executor used to just break but this didnt prevent it from downloading all other images stored in my array even tho they aren’t saved on the disc, this caused the scraper to wait indefinitely for a long time, since it now has to fetch double or sometimes more the amount, but I have now fixed that

Next steps?

I will now clean the dataset, and since by cleaning the dataset I am actively contributing to the completion of the project, I have got permission to use lapse to record that time