5 days plan ready

Self-fish devlog #11 (Very Important)

So in the last devlog I had completed the add item screen, since then I have been making plans for the future of this app, and i think I am finally ready to reveal it.

What have I done?

• Added tiers and a ration norm to the item and detection table, I will explain later in this devlog, what they do
• Made the whole backend more secure from wrong answers generated by gemini: since I am using gemini, there is a possiblity that it does not follow the schema I gave it, if that happens, the backend would crash, but now have added field validators for all crucial columns where the the model is most likely to generate wrong answers.
• While testing I also found out that at sometimes, the model would face a lot of traffic and it would return a 503 and the server crashed, but now the backend has 2 models, gemini-3.1-flash-lite and gemini-2.5-flash-lite, each with 5 retries. So if gemini-3.1-flash-lite, which was getting a lot of traffic, fails, the backend fallsback to 2.5, which I believe should have less traffic
• Also in the last devlog I had forgot the add tag button which I have added, I already had the function to add but I think the 25 day gap made me forgot that, I also added tiers to page but I still haven’t added ration_norm cuz I have to devlog first

What’s the plan?

A lot of people don’t actually know what communism is, so I will explain it first, communism is a stateless, moneyless and classless society where the people own the methods of production, It is a type of socialism that came into existence through Karl Marx

So in true communist state, there is no money, therefore I dropped the price column from my items table.

Instead I have divided everything into 3 categories:

**1. Essentials - These items are basic necessities that are needed to surve, example: food, water, soap, clothes, etc.
These would be distributed equally to everyone who uses the app, ie the people of the state.

**2. Useful Items - These items are not necessary but they improve your quality of life, these include things like a normal mouse, a laptop, etc. These are not distributed equally, instead you have to write 2 lines, just 2 cuz I know when others would review my project, they wouldn’t want to write a whole para to check the working of my app, so those 2 lines would go to gemini once again and it would check if the person gets the item or not.

3. Wants - These items are not necessary and do not improve the quality of life, but they are important to satisfy the personal wishes of the user. These include top tier laptops, high end gaming mouses etc. For this I want to connect to hackatime, I had seen some project on stardance that connected to hackatime, so the plan would be to connect to hackatime and get the no. of hours coded by the user. These hours would be used as labor hours. Whenever the user wants to get these items with the want tier, they would give some of their labor hours to get their item along 2 lines to satisfy the model.

So that’s how the app would work, I have just finished the first part of the app, this probably is the 2nd hardest part of the project, the next step would be to create a page where all the items are listed, this would be very easy but after that would be the hardest part of the project, it would be the form where the user can request the item.

The images:

Well they are both the same, As you can see I have Tiers and add button to this page, everything else is the same,
there are 3 buttons (pressables) with the tiers, the default tier is selected by gemini, the user can change them, The ‘+’ button would be present next to the last tag, It can toggle adding, the user can then enter the tag name and press the + button next to the input field. The section below the tag selector displays the description of the tags.

I spent 10 hrs on single page (screen)

Shelf-fish Devlog # 10

I didn’t knew it would take that long

So the add item page took around 10 hours to make, more than 10 hours if we include the time I spent 25 days ago to make it work, so here is what I have done:

• Completely change the design of the page, the previous design really sucked, It was there just so to make everything work, my previous self, for some reason, didn’t care about the design

• The new design uses something similar to drop-shadow except there is no box or drop/box shadow in native-wind, so i had to come up with a way to make my own, i used a self-closing View tag with absolute positioning, no inset and translated to some ammount with a background color…

• This offset View works as a drop shadow or box, it really doesn’t make difference here as I am working with rectangles. The Offset component worked when I using with a static color and size but for some parts of the design I needed different colors of shadow like the item image which uses red shadow or the tags which require small shadows as they themselves are pretty small. I made it dynamic by using inline css cuz native-wind calculates colors statically

• Though for some parts I didn’t used the component but used another View in the same way as the component cuz I was frustated and wanted flexibility which the component lacked at that time.

• The whole app now has a communist theme as explained in the previous devlog

• I have switched from using native-wind colors to using my own, these are the same i used for my earlier project

• I have also changed how inputs look, they really sucked previously as they didn’t had titles and the design was low effort too, now the title sits above the main input field and the input field has a sharp look with borders and drop-shadow

• The tags took the most amount of time, I tried so many designs, but all looked very repetitive with the inputs and I wanted them to fit in too. Also I needed to have 2 designs for the tags, as i have a button to toggle removing, when not removing the tags are small and cute but when you toggle removing, they become big so that it is easier to touch them.

• The final add button also took a lot of time, the same issue as the tags, since it is the main call-to-action, it needs to be unique, so I had to make it different, It incorporates all the three colors in the design, black, gold and black along with the communist logo. It also has 2 states, the normal state is what is in the attached screenshot, but when it is submitting the add request to the backend, it sticks to its shadow by translating to its shadow, i mean it just animation

That’s it for this devlog, since the last devlog was very short, I had to compensate for that and I over-explained in this one

The next steps would probably be to connect the detecting code to route to this page with the item object, and also work on the main distribution part which i think would be the hardest.

Where I’ve been

Soooooo It’s been 25 days since my last devlog, that means I haven’t coded for 25 days, the culprit for this is my exams, I literally have forgotten where I was in the project, also the hours associated with this devlog are also 25 days old, soo this devlog won’t tell you much, the only thing i remember is that I was making the add item page and it was working, i just had to do the ui for it.

My next steps would be to figure out where I was and create some kind of plan to complete the project also since an inventory management app would not be unique, I want to give it my touch and make it communist, I would try to incorporate accuratee communist economy make this the inventory of the people.

Ooopppss I exposed jwt tokens in github

Shelf-fish Devlog #8

Yepp I exposed my jwt tokens in my github, but trust me I knew what I was doing, those tokens are long expired, my backend isn’t published yet and i will change the exposed passwords, which was 12345678, cuz why not

Anyways I have done a lot things, in the last devlog, i had connected the frontend with the backend and made the detection endpoint work with the frontend. i had to manually type the authentication tokens cuz i had not set up auth in the frontend, but I have now fixed this

Here’s what I have done:

• Created tokenStorage.js - it handles getting the tokens, saving them and clearing them using expo-secure-store, i used expo-secure-store cuz it saves tokens in keychains rather than files that makes them much less likely to leak or read by anyone/app/malwares
• Created authService.js- this one handles send the get and post requests for registering, logging in and getting current user data
• Added interceptors to Api, i am using a single api instance using axios.create() across the app, so i have added interceptors to attach auth tokens and check if they are valid or expired
• Created the login and register screens, they don’t look good now but will work on the ui after i get everything set up, I will make everything exist first, the designs can be taken care of afterwards
• I also created a new api route at ‘/users’, this one is responsible for getting user data, right now it just gets the user info but I think i would also make it update it…
• I also added navigation using react-navigation

What’s next ?

• I haven’t finished the home screen yet, well nothing is finished yet, but the home screen just greets the user, i will add buttons to navigate to camera and upload components
• I will also create the item related pages, that will show all items and let the user create items after detecting the product

The images:

They look soo bad but as I said I will make everything exist first and then make the ui good

1. The register page
2. The login page, it’s not even filled yet, cuz i want to add text and a logo, i will create a logo too lol i forgot about that
3. The home page, empty page with a greeting and a sign out button, it was made to check if everything is working or not

Frontend Connected

Shelf-fish devlog # 7

In the last devlog, I had finished creating the detection logic using gemini api, after that I have done a lot of things and suffered a lot of headaches too

What have I done??

• Created api endpoints for detections at /detections and added it to the fastapi app.
• Added authentication dependency to the detection api so that only authenticated users can upload images and we can know who detects which item.
• Created the Camera and Upload image logic, I have used expo-camera and expo-imagepicker for that, it was pretty easy, expo’s docs are straight forward and I have used them before
• Created the uploading function that will send a post request containing the image to the backend.
• I also added storage fn to upload the captured/picked image to cloud using cloudinary.

What’s next?

• Now I will work on the ui and create login and register screens cuz manually typing the authentication token to the uploading fn is really frustrating
• I also have an idea for this project which would make it very unique cuz at this point it just looks like just any other inventory management systems.

The images-

1. console log of the item after it was detected by gemini and saved to database,
2. the image being displayed after clicking the url,
3. the images in cloudinary, the other one is stock image and was there automatically.

Detection Working!!!

Shelf-fish Devlog #6

TL;DR

What have done?

• Created a basic jwt auth
• Used Gemini Api and Gemini 3.1 Flash Lite to create the product detection logic

What’s next?

• I will now make api routes for detection that would output the brand, item name and category of the item

Got lots of time?

Sooo the auth and detection logic is complete, we are now close to ending the backend, earlier after dropping the plan of using a classification model, I had decided to use an ocr to detect text from the input photo and then use that text as an input for openai api and ask it to identify the brand name, item name and category from the image, but since I am broke , I couldn’t use openai api as it costs money and then learned that gemini has free api plans that include multi-modal inputs like images and give out text, so I swited to that and it works flawlessly, the rate limits are also generous, at 500 requests per day and 15 per minute. I tested it with 6 text images and it gave 100% acc but the sample size is pretty small to say that it is 100% accurate, but it does work.

The detection logic, at this point, is just a function, i need to make api routes for that and add a detection log table, the log table would be useful at providing data about rate limits and is also nice to have if the project get bigger, I also have to make a list of categories out of which gemini would choose the category of the product cuz at this point, it outputs anything it likes as a category

Backend is Up

Shelf-fish Devlog #5

What have I done??

• Set up FastAPI and SQLModel - I have successfully connected and set up the backend with the database, I am using neon’s postgres db

• Created Item table - I also created the first table of item, it has the following columns- Name, id, quantity, category and price

• Set up CRUD - set up basic crud end points for the item table

What’s next??

• I will now create auth using pyjwt and passlib
• Next I will work on the core mechanism for extracting name from the product image using ocr

Dropping the plan

Shelf-fish Devlog #4

Heyy gng, since the past few days I have been just editing and reviewing images and they don’t seem to finish any time soon but stardance is finishing soon and I want to finish this project before that so I am dropping the workplan of using a classification model for identifying items

That approach was flawed, I had not thought about that earlier instead I was biased by my previous successful project using the classification approach to identify pokemon

I think a better approach would be to use an ocr and then ask an llm to parse that into brand name, item name and weight and other things

Despite the my orignal plan failing, I want to highlight that engineering, in all forms, is itterative, no one finds the solution to the problem in their first try and the things I have made before, ie- my scraper, is not obselete at all, it works so good, like it scraped 150k images in 6 hours, that’s really insane and very useful too, I will make it as another seperate project in the future cuz I think it deserves some more recognition and will improve it too

first 3 images- images of clean dataset, products are cropped and all noises are deleted
last image - for comparison, an unclean dataset, a lot of noises in it like people and some images dont even have the product

Who is Who - Shelf-Fish devlog #3

There were so many duplicate images

So in the last devlog, I told my scraper scraped nearly 150k images, but when I began cleaning them I noticed a lot of images were exact duplicates, I thought it was just for a class or two but It was for every single one of them , except Oreo

After manually cleaning 14 classes with duplicates, I gave up, and decided to use python to delete duplicates, I used hashlib and md5 hash to detect exact duplicate images inside all folders of my root train images folder which each represent one class, and I used os.remove to delete those duplicates.

I ended up deleting around 34k images which brings my dataset from 150k to 100k images, but this would change too cuz the 100k maybe unique but they still have a lot of discrepencies, like random products and all.

I also used matplotlib to plot a graph of the images remaining, and the results dont look good for a few classes which have now less than 300 images, others are in range 500 - 800, remember they were around 1100 images earlier, so around 800 images were duplicates of the remaining 300.

I know what’s causing this, multiple search queries, since google images only displays a fixed 400 images per search, I made my scraper in a way that it searches for the same product multiple times but with a different query which causes popular images to be duplicates

One more thing, the dataset would not only decrease in number, but after cleaning the images, I would augment them which would drastically increase the number of images and also make the model less likely to overfit.

Also the graph is really ugly cuz there are like 130 classes on the x axis and they dont really fit well and I am not an expert in pyplot so I dont know how to make it look good

Shelf-Fish Devlog #2

My Scraper is on steroids !!!

I scraped 150,000 images

Yooooo guys the scraper worked too good now, it scraped roughly 150,000 images in just 6 hours, that is insane, the speed was around 7 images per second or 420 images per minute, that is really crazy, my old scraper took a full day for 75k images, this is like a million times faster!!!

In the last devlog, I acheived a speed of around 1079 images in 10 minutes or around 100 images per hour, so since then I have made it 4x times faster.

Here’s what I did:

• Added premature scrolling - earlier the scraper would miss a lot of images or would wait for a long period of time for the images to load and then scroll, but now as soon as the main div appears on the screen, the scraper scrolls down a bit triggering lazy load instantly and then scrolls down to the bottom triggering te lazy load 2 times in a cycle, this made the whole scrolling thing so much quicker.

• Fixed downloads not cancelling - I am storing some extra urls so that if one or more of the urls failed to download, I have backup for completing the target, but I am using a ThreadPoolExecutor to download images concurrently, when it reaches the target the executor used to just break but this didnt prevent it from downloading all other images stored in my array even tho they aren’t saved on the disc, this caused the scraper to wait indefinitely for a long time, since it now has to fetch double or sometimes more the amount, but I have now fixed that

Next steps?

I will now clean the dataset, and since by cleaning the dataset I am actively contributing to the completion of the project, I have got permission to use lapse to record that time

Shelf-Fish Devlog #1

What is Shelf-Fish?

It is a computer vision based cross platform inventory management and billing system that I am making with react native as frontend and probably a fastapi backend.

How will the project work?

The project would be made in several phases that are listed below:

• Phase 01 - Scraper

  In this phase I will make a fast scraper with selenium that would scrape 1000-1500 images for over 130 grocery items that I have listed.

• Phase 02 - Training

  In this phase I would clean the dataset and train the model!!

• Phase 03 - Backend

  I would build a basic cors backend in fastapi and add some user relations

• Phase 04 - Frontend

  This phase involves building a cross platform fronend with react native.

• Phase 05 - ESP 32

  This one is not confirmed yet, but if i get the time, instead of using a phone, I would use the esp 32 and esp32 cam to register objects and create bills, or perhaps I should make this the main focus of the project and make this a hardware project, I am still not sure.

What have I done till now?

I am modifying my old scraper that I used for my previous pokedex project, which was very very slow, I have changed it completely, It now directly goes to the image search url instead of going to google.images.com and then manually typing the words. It now also downloads images concurrently using the ThreadPoolExecutor, earlier this part was the slowest but in just 2 seconds, it downloads 125 images,

But it is still pretty slow, It took over 10 min to scrape 1079 images, and it was a bit short of the 1125 target images which is due to slow network that caused the images to not load. For the next devlog I will try to make it sub 8 min or faster