DEVLOG #6

I have been working on the leads route for the past 3 days and this has been a lot of slow progress. I have been coding less and more thinking about the design of how leads are generated. Basically, the 2 endpoints that I have made till now are /leads and /leads/generate-leads.

`/leads:`

This endpoint is responsible for simply getting all the leads generated for the team. It simply takes the current user, finds their team and query the database to find all the leads for that specific team ID. This was pretty simple to make and there is not much to describe in this.

`/leads/generate-leads:`

This is the endpoint that took a lot of iteration and design changing to get right. The basic idea that I had before making this endpoint was that I use my existing scraper function to scrape the businesses, then check the database to see if the lead already exists (this is the point which take a lot of time to design) and then either add the lead or not. Now it was simple in theory, but there were a lot of questions like - what is the parameter to check if a lead exists?, If a lead exists then should the newly generated lead just be dropped?, if we drop the lead then what about the case when something about a business genuinely changed like its name or something?, what if the user wants to still act on duplicate leads?, etc.

I will not go in detail about all the failed attempts at designing the structure for lead generation and I will instead just focus on what design I finally decided.

It is basically a flag system which flags duplicate leads and the type of duplicate.

we make sure that the user requesting for lead generation is a manager of their team and not an employee (cause employees don’t get to generate leads for the team).
we use the scraper to generate the leads.
we loop through the newly generated leads. In each pass, we query the database to find if there is an existing lead. Now, I realized that a lead is basically just a phone number. Thus, if a phone number for that specific team already exists in the table then we have a duplicate lead. If we do find a duplicate lead then we move onto the Lead Sorting Logic. Else, we simply add the lead to the table as a fresh lead.
Finally we commit all the changes to the leads table after the loop completes.

Lead Sorting Logic

If name and website changed -> Flag = 0
If only name changed -> Flag = 1
If only website changed -> Flag = 2
if completely duplicate (nothing changed) -> flag = 3

This sorting makes it easy for the UI to display duplicate leads to the user and I am planning on adding a feature where the user can decide if they want to reinstate the duplicate leads for assigning quota.

Next, I will be working on some bugs that I think are in my current logic. Right now, I am asking the user to prematurely enter the number of leads they want to generate without knowing how many businesses were even scraped for their query. For example, if someone wants 100 leads for their query but we could only scrape 80 then there will be an error according to the current logic. Therefore, I am thinking to split the lead generation into 2 parts. First the scraping part and then the generating part. For that, I simply need to restructure my single scraper function into 2 separate functions. Then I will call the 2 functions one by one in 2 endpoints.