Self-Fish Devlog #2

My Scraper is on steroids !!!

I scraped 150,000 images

Yooooo guys the scraper worked too good now, it scraped roughly 150,000 images in just 6 hours, that is insane, the speed was around 7 images per second or 420 images per minute, that is really crazy, my old scraper took a full day for 75k images, this is like a million times faster!!!

In the last devlog, I acheived a speed of around 1079 images in 10 minutes or around 100 images per hour, so since then I have made it 4x times faster.

Here’s what I did:

• Added premature scrolling - earlier the scraper would miss a lot of images or would wait for a long period of time for the images to load and then scroll, but now as soon as the main div appears on the screen, the scraper scrolls down a bit triggering lazy load instantly and then scrolls down to the bottom triggering te lazy load 2 times in a cycle, this made the whole scrolling thing so much quicker.

• Fixed downloads not cancelling - I am storing some extra urls so that if one or more of the urls failed to download, I have backup for completing the target, but I am using a ThreadPoolExecutor to download images concurrently, when it reaches the target the executor used to just break but this didnt prevent it from downloading all other images stored in my array even tho they aren’t saved on the disc, this caused the scraper to wait indefinitely for a long time, since it now has to fetch double or sometimes more the amount, but I have now fixed that

Next steps?

I will now clean the dataset, and since by cleaning the dataset I am actively contributing to the completion of the project, I have got permission to use lapse to record that time

Shelf-Fish Devlog #1

What is Shelf-Fish?

It is a computer vision based cross platform inventory management and billing system that I am making with react native as frontend and probably a fastapi backend.

How will the project work?

The project would be made in several phases that are listed below:

• Phase 01 - Scraper

  In this phase I will make a fast scraper with selenium that would scrape 1000-1500 images for over 130 grocery items that I have listed.

• Phase 02 - Training

  In this phase I would clean the dataset and train the model!!

• Phase 03 - Backend

  I would build a basic cors backend in fastapi and add some user relations

• Phase 04 - Frontend

  This phase involves building a cross platform fronend with react native.

• Phase 05 - ESP 32

  This one is not confirmed yet, but if i get the time, instead of using a phone, I would use the esp 32 and esp32 cam to register objects and create bills, or perhaps I should make this the main focus of the project and make this a hardware project, I am still not sure.

What have I done till now?

I am modifying my old scraper that I used for my previous pokedex project, which was very very slow, I have changed it completely, It now directly goes to the image search url instead of going to google.images.com and then manually typing the words. It now also downloads images concurrently using the ThreadPoolExecutor, earlier this part was the slowest but in just 2 seconds, it downloads 125 images,

But it is still pretty slow, It took over 10 min to scrape 1079 images, and it was a bit short of the 1125 target images which is due to slow network that caused the images to not load. For the next devlog I will try to make it sub 8 min or faster