DEVLOG #7

ㅤ

SCRAPER

Made a few significant changes to the scraper:

1. Earlier, my scraper was based on the CLI logic where the user will enter their query and then the program will scrape all the businesses for that specific query before asking the user about how many leads they wanted to generate. But, now that I am switching from CLI to web, this logic is broken (even from a CLI perspective this logic was broken). Basically, if there are 124 businesses found for a certain query then the user would have to wait at least 84 seconds (I tested this time and it is not an estimation) for the whole search process and generation. Now this long wait time is because the program is basically asking the user the limit of leads they want to generate but the point of application of that limit is incorrect. For example, if a user only wants 50 leads, then the program would still go through all the 124 businesses and then only generate 50 leads out of those 124. Thus, all the searching and scrolling through the menu takes time. Whereas, now If we instead make it such that the user inputs an upper limit value to their search all along then the time would be significantly improved. For example, If a user only wants 50 leads out of however many there are, we now don’t try to find all the leads and then apply the limit, we instead apply the limit straight at the searching step and then generate all the found leads. Using the same 124 total leads example, now the scraping and lead generation time is only 33 seconds.

2. while randomly talking about my project with gemini (and glazing it), I suddenly got a feedback for it out of nowhere. It said “wow, its great that you are using playwright. Make sure to not make new browsers for each user or else your scraper will burst cause of the ram usage”. This completely unexpected feedback really made me think cause that is exactly what I was doing up until now. So, I asked it how to fix it. It told me to create a single browser for the lifetime for the app and for each user, simply make a new browser.context(). Well, I was going to do that but I found out that to do that, I first have to turn my whole playwright script into async. I do remember why I had to do it but if I explain that right now then it would take too much time and would be too boring. The simplest explanation that I can give is that sync runs on multiple threads, so if 2 requests coincide, the data could get merged or damaged. Hence, async is like the ultra time managing chef which works on a single thread but manages all the coming requests so that it works even when it is waiting for another process to happen. So, after doing all of that, I think my scraper will not blow up now. This was a precautionary step more than an immediate necessity but its good to handle it early than to regret not adding it after the project goes live.

/lead/reinstate ENDPOINT:

I also made the reinstate endpoint which acts on duplicate leads. There are 4 specific cases that it looks for:

1. If user provides a flag value -> reinstate all leads with that flag by clearing the flag and pushing changes.
2. If user provides a lead ID -> reinstate the lead with that lead ID by clearing the flag and pushing changes.
3. If none are provided -> reinstate all duplicate leads.

By default, a user cannot pass both flag and lead_id.

---

Now that all of my basic endpoints are complete with all the tables covered and linked together successfully. I can now move onto the more deeper logic of the program and work on some more features like call records with the ability for the users to mark the status of the call, etc. and other stuff that I have thought of. Also, I made a new frontend.md to store all my fronted ideas as I make the API and ideas.md as I come up with more useful ideas.

commit: 2fbba880236ed9644733666de2cfb4dcacba40cb