DEVLOG #4

I reformed the Signup API flow and also made the Login endpoint.

The new signup API flow goes something like this:

PAGE-1:

1. Enter full name
2. Enter email
3. set password
4. Submit button
   when submit button is clicked, we query /users/check-email to check if the account already exists. If account exists, the API simply returns a status_code=400, detail="email taken" else it returns {"available": True}.

PAGE-2:

1. Set Username
   When submit button is clicked, we query /users/check-username to check if the username is available or not. If username is already taken, the API simply returns a status_code=400, detail="username taken" else it returns {"available": True}

PAGE-3:

1. Choose if signing up as a business or as an individual.

PAGE-4:

1. Set team name if applicable.

Finally, we have all the required data and data validation to make a new user so we can POST to /users to make a new user. Now, I had to think about this a bit because what if a there is a scenario that while a user is still making their account, they enter a username and go to the on-boarding step with the team name entering and all that stuff. And meanwhile, another user also enters the same username and creates the account by finishing the on-boarding before the first user who has the same username but did not create the account yet? when the user who is currently on on-boarding finishes all the steps, the server errors if there is no second data validation with all the collected data. Hence, I added validation of username and email in the second step too so that this case is handled correctly.

I also made the simple login endpoint which takes the email and password and validates the user and their hashed password. I am planning to keep the login a single page and hence there are no live validations I need to make. Thus, everything lives in a single /login endpoint.

Next Steps

I am also wondering that in the API response, I do not signify anywhere that if the user is currently logged in or not and I do not know how to keep the user logged in until either they sign out or something else so I need to learn that state management too. I will work on that next.

DEVLOG #3

Today I worked on just the signup endpoint and the password hashing. I basically made an endpoint which would take all the collected data: full_name, username, email, hashed_password, role, team_name and use it to create team and user on the database. I have successfully implemented the basic idea that I had but it currently has no error handling like duplicate user checking or existing username.

So, what I thought was to keep a single endpoint and do the data validation at the end when the api call is made by querying the database first to check for unique email and username. But, That was not good user flow according to me, and I was looking for something like this:

PAGE-1:

1. Enter full name
2. Enter email
3. set password
4. Submit button
   Then we will internally check if the email exists or not. If it exists show “account already exists, login instead”. If the email does not exist then move into the next page.

PAGE-2:

1. Set Username
   Check if username exists. If username already exists, show the error “username already taken” else move onto the next page.

PAGE-3:

1. Choose if signing up as a business or as an individual.

PAGE-4:

1. Set team name if applicable.

All the data required for the database entry for user creation is complete so after a few more random onboarding steps, commit all the collected data to the database to make the team and the user.

Basically this is the whole signup flow that I want with the most basic data collected from the user initially. Now what I want it some way to check the validity live on every page and I do not know how to do that. I will need to display the error messages and all through JS though I don’t know if I can also query the database live through JS to do some live validation. I will not learn this FastAPI concept and try to implement it in my API

DEVLOG #2

Created all the database models along with database.py with the boilerplate code for making database and tables.

Database models that I made:

1. Team -> id, team_name, created_at
2. User -> id, full_name, username, email, hashed_password, role, team_id, solo_team_id, created_at
3. Lead -> id, team_id, name, phone_number, website, rating, created_at

I also made some changes to scraper.py by adding more filtering such that:

• If business does not have any phone number but has a website, lead is added.
• If business has a phone number but no website, lead still added.
• if business does not have any phone number or website, it is basically useless and excluded.

I have not faced any major challenges yet as it has mostly been boilerplate code straight from the docs or simple repetitive code. To be honest, this has been boring. But, now I will be starting with the FastAPI stuff and actually start making the API and I am very excited for that.

DEVLOG #1

The scraper is something that I had completed a few months back. I made it for my friend’s business. Now, I am trying to expand the scope of this from just a scraper to a full Business platform where a whole team can collaborate on cold-calling with daily quotas and a manager who can view stats about his employees. I also want to make a live chat option to better increase coordination between the team. It is a pretty basic project but a pretty ambitious for me.

Right now, I have completed all the error handling and re-enforcing the scraper logic in the 4 hours that I have spent, to make it ready for a web-UI.

I will now start working on making the actual API and database for my project.

There are 3 basic tables that I have identified:

1. Teams
2. Users
3. Leads

I believe that all other features can build on top of these 3 as these are the basic building blocks. Hence, I will start working on making these right now.

This is my first time working with FastAPI (earlier I had only used Flask) so there will be a learning gap but I have experience with Flask so I think that transitioning would be easier.

DEVLOG #21

Completed the CLI tool and created binaries for windows and linux using pyinstaller.

Also completed the full Guide page on the website with all the relevant rules, syntax and limitations listed cleanly.

DEVLOG #20

Completed and wired Unparser

TRY NEW UPDATE

I completed the unparser now with the precedence based approach and I can’t believe how much simpler it is and how bad my initial approach was.

The thing that makes it work is the new needs_paren function which basically does all the precedence based logic of telling if the child needs parenthesis or not. Basically if the child’s precedence is lower than the parent op then we need a parenthesis. For example: if the child is a + node and the parent is a * node then we would need parenthesis on the child. For example:
(x+1) * 3 -> x+1 needed parenthesis.

The main part that made me think a bit was the case when precedence of the child and parent is equal. There are basically 3 cases here:

1. When op is + or *, the child never needs parenthesis.

2. When op is - or /, the child only need parenthesis if it is on the right side. For example:
   precedence of + == -, a + b - c -> a+b is on the left side and hence does not needs parenthesis. Whereas, c - (a+b) -> a+b on the right side needs parenthesis because this basically means c - a - b and not c - a + b. Hence, parenthesis is required.

3. When op is ^, parenthesis is only required when the child is on the left and not when it is on the right. For example:
   a^b^c does not need parenthesis (even though it could be written as a^(b^c)) but, (a^b)^c needs parenthesis because this means a^(b*c) as the powers multiply.

I have also wired in the unparser function to derivative and simplify too so now the result is an expression instead of a tuple.

Now I will update my README.md and explain the working of the unparser and then I will work on adding the guide page to my website.

DEVLOG #19

Another Setback

I was working on the unparse function today. Basically the idea was simply to recurse through the tree and form the expression as a string. Now, the hard part in this was to decide when to apply parenthesis. Basically, we have to apply parenthesis to any child of the function which has a lower precedence then the current parent op. I did not have this idea as well set in my mind when i was making this function and I just wasted 4 hours today trying to stack up elif cases with or conditions to try to cover every shape.
‎ ‎ ‎
Basically, I was thinking that for *//, there are only 4 base cases:

1. both children need parenthesis
2. left needs parenthesis
3. right needs parenthesis
4. none need parenthesis
   ‎ ‎ ‎
   I took that as a starting point and then i started to think of cases which would satisfy that and that caused me to build up this or-or-or jargon. I am really pissed.
   ‎ ‎ ‎
   Now I will change my direction and move onto a precedence based approach which checks if the precedence of the child is lesser than the parent op, if yes then parenthesize it and if no then check if precedence is equal, if it is then check for associativity. That is the plan and I hope I don’t mess up this time.

DEVLOG #18

LIVE PROJECT

I have finally deployed the Version-1 of the project on vercel, check it out at the link above.

I also completed the README for my repo explaining all the features and limitations of the project. Now I will be working on making an unparser to show the users a clean expressions as a result from the calculator instead of AST nodes and I will also add a guide page on the website for the syntax rules and general guide.

DEVLOG #17

Completed Styling

(would appreciate feedback)
‎ ‎ ‎

Features:

1. AST generator
2. AST visualizer (downloadable SVG trees)
3. Domain calculator
4. Range Calculator
5. Derivative calculator
6. Simplification
   ‎ ‎ ‎

Next steps:

1. AST back to expression generator because right now all the calculators give the result in terms of AST nodes so i need a reverse-parser which I think would be easy.
2. Making a simpler bare bones CLI tool for simpler People.
3. adding a guide page to the website explaining all the limitations and syntax of the calculator.
4. hosting on my own domain?

That is all that I can think for now.

Right now I have not hosted the website because I am thinking of adding a few more features like user changeable SVG node and line colors, etc.

DEVLOG #16

More PROGRESS!

1. completed range.py

2. tested edge cases with some pretty nasty ones and have decided to leave some limitations like not supporting gif/frac/sec/tan… functions because of their nature.

3. General Debugging

4. created all the flask endpoints along with the basic HTML layout of the website and wiring in the JS.

5. Website is tested and all endpoints are working

Now, I will focus on adding some styling to the website and also making the user experience better.

DEVLOG #15

Real PROGRESS!

1. Completed the full differentiation pipeline which is working completely now with all edge cases (that I have tested) handled.
   ‎ ‎ ‎
2. converted simplify to a standalone file as I noticed that simplify could be used as a general simplification pipeline for ALL generated nodes in my maths suite.
   ‎ ‎ ‎
3. Created pow_to_div function in simplify to turn negative exponents into division nodes to make the simplification pipeline truly generalized because none of my other algorithms expect a negative exponent.
   ‎ ‎ ‎
4. A LOT of debugging in domain.py and testing some edge cases.
   ‎ ‎ ‎
5. Adding support for multiple exclusions instead of single in normalize_domain.
   Earlier: (-∞, -2) ∪ (-2, 2) ∪ (2, ∞)
   Now: R-{-2, 2}
   which is a much cleaner solution
   ‎ ‎

Now I am working on completing range.py and then finally making the website for the full maths suite.
‎ ‎ ‎
I am a bit confused. I am not able choose between flask and FastAPI. If you have read all this then would you also mind sharing your opinion on what should I use in the comments?

DEVLOG #14

‎ ‎ ‎ ‎ ‎ ‎ ‎
Completed the full simplify_initial pipeline. Handled cases with 0 and 1 and traced a full long recursion by hand to better understand the pattern and make iterative debugging faster. Also handled negative node by simplifying the ('*', '-1', right) node formed so that any coefficients collapse cleanly and the term can form correctly with a negative coefficient.
‎ ‎ ‎ ‎ ‎ ‎

EXAMPLE:-

• Earlier: x - 3x -> terms = [(1.0, 'x', 1.0), (-1.0, ('*', '3', 'x'), 1.0)]
• Now: x - 3x -> terms = [(1.0, 'x', 1.0), (-3.0, 'x', 1.0)]
  ‎ ‎ ‎
  Therefore, now merging terms will result in ('*', '-2', 'x')

I pasted my code into claude and asked it to generate me some test cases which it believes would fail and now I am working on debugging them with a mix of hand tracing recursion and simple fixes so the pipeline is almost at its end (I hope).

DEVLOG #13

‎ ‎ ‎ ‎ ‎
I have completed the basic simplification functions now but, it is not working perfectly and there are still some changes to be made. for example, handling cases where it is multiplied by 0, or when 0 is added, or it is multiplied by 1, etc. so I think it requires a bit more of iterative debugging to catch all of the edge cases. I have formalized the design of simplification into: Flatten -> form_term -> merge_terms -> form_node -> rebuild‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎
which is much better then the previous logic that I had where I was adding more edge cases for handling each case.
‎ ‎ ‎ ‎ ‎
Right now I will be testing the full recursion for a test case to check where it goes wrong right now in the pipeline to fix it.

STUCK

So basically, since my last devlog, the main feature of the new derivative calculator that I had proudly announced is now coming to bite me. The simplification is so tough. Let me explain you all the turn of events. (Simplification is still not complete)
‎ ‎ ‎ ‎‎ ‎
basically, my initial idea of simplification was just removing all the temporary zeros and ones that were generated by the derivative itself. Then I thought of increasing the initial scope to support some other simplifications like (‘’, ‘2’, (’’, ‘2’, ‘x’)) should be solved to (‘’, ‘4’, ‘x’). I managed to achieve this simplification too but, I found out there are a lot more simplification cases that I need to handle after we get the derivative. for example: x - 3x, which should be simplified to -2x. For that I had to build an approach which would take the coefficient and base then operate on the coefficients
for the same bases according to the operator. Therefore, for x the coeff is 1 and then for 3x the coeff is 3 but that is incorrect. In the case of x-3x, the coefficients should be 1 and -3 so that when i add them i get -2 as the final coefficient for x. To do that, I had to switch all binary subtraction expression and turn them into additive unary negative for op = ‘+’ so that x - 3x turns into x + (-3x). Now, all this actually worked but the main problem was that now the flattened list looks like ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ [‘x’, (’’, ‘-1’, (‘’, ‘3’, ‘x’))] which means I cannot reliably extract the coefficients now. I first have
to handle the recursive case of flattening the node again, turning (’’, ‘-1’, (‘’, ‘3’, ‘x’)) to the flattened version
[‘-1’, ‘3’, ‘x’] and then i would have to run it back through combine where it would be formed as calc = -3 and symbols = ‘x’ and then rebuild it as the node (’’, ‘-3’, ‘x’). Only after that can i actually extract the coefficients and compare the bases of the two expressions which would be 1 and -3 and calculate them according to the op => ‘+’ to finally give -2 as calc and ‘x’ as symbols and then rebuild that again into (‘’, ‘-2’, ‘x’) to solve it.
‎ ‎ ‎ ‎
Now this was just one case for one operator but i need to handle both ‘+’ and ’’ as the both have associativity and there are other cases for other ops too like x^2/x should just simplify to x. I have already added so much code that I myself feel like the approach is very wrong and the more I build the more edge cases I am adding because right now the amount of if-else handling for specific cases is crazy so I really think that I need to reconsider my approach for simplification. The derivative calculator is taking too much time considering it is an intermediate step but I think it would be worth it because this simplification step is something that I have never done before so I would be learning something anyway.

Derivative

For finding the range, I am opting for method of differentiation to check intervals. hence, I have made derivative.py for differentiation.
‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎ ‎
This is my second derivative calculator. The first one that I built was made completely using string slicing and weird string manipulation and pretty biased logic where everything was stored in the form of a dictionary. To be honest, I was pretty proud of that calculator because I was able to make it just using the knowledge I had, and recursion is not innate (I think) and I was still able to do it so I am proud of it. But, it was no doubt pretty inefficient with a lot of edge cases. Therefore, this time i am making a better calculator using my new AST representation and A LOT more recursion. This calculator will process more functions and it will be able to simplify the derivative too, for example: derivative of 2x^2 will not be displayed as 2*2x, rather it will be correctly displayed as 4x.

Devlog #10

Completed domain.py

finished coding domain.py and tested about 20 test-cases on it. Completed debugging to finally finish domain calculation and start with Range.

I do not know what else to tell you all as it has been a few days of coding and debugging with different bugs surfacing like incorrect handling of undefined functions, normalizing output, handling union cases correctly, etc.

Union and Intersection of intervals

I have made the intersection function which takes intervals and unions them together. I am now working on the union function. I initially used recursion to do the job but that was incorrect that the better method was just to sort the intervals list and then use a simple loop to evaluate the union. Here is the issue i was facing:

[[A, B, C, D], E]
AUE-> False, BUE-> False, CUE-> True => [F, A, B, D]
union [F, A, B, D]-> FUA-> False => [[F,A], B, D]
again [F,A], B => FUB-> False, BUA-> False => [F, A, B] => union this list now => FUA-> False => [[F,A], B]
then again [F,A], B => FUB-> False, BUA-> False => [F, A, B] => FUA-> False => [[F,A], B]
this goes on and on infinitely
This is a problem of infinite recursion

It has been a tough journey

If you read my previous devlog, it almost feels like i was foreshadowing what was about to happen to me.

While making the solve_node function which would solve the inequality for x, i got lost in the mess of symbolic mathematics and started handling cases with fallbacks upon fallbacks. It has really a hot mess, I got to find out that there can be many types of questions and to solve every type of domain question, I would need fallback logic for every type of question which would mean I would have to encompass all the techniques in domain finding into the file and manage the interaction of the techniques and orchestrate them at a high level and basically making my own sympy (which took decades to build).

I even thought of quitting a few times and the project was basically at a halt for 1-2 days. I then decided to solve a simple domain question by hand to check what really are the steps of finding the domain of a function. I found out that my logic up until making the constraints was fine. The next steps were finding critical points for example: (x-2)(x+3)>0 would have critical points 2 and -3 (they are just the points where the function is zero). Now, I also know that critical points also occur when the sign of the function changes or when its nature changes (going from defined to undefined and vice-versa) so instead of making a symbolic critical point finding function I thought of a more brute force method.

If I have a number line (range of numbers) and i simply walk through them one by one, noting sign change between 2 numbers as a sign “there is a critical point here” and then use another function to basically go into each of those marked intervals and continuously half the interval (think of it as taking an interval and then zooming into the interval until you find the critical point) until about 9 decimal places (could be anything, it is just the amount of accuracy), you would get the critical point pretty accurately without having to solve anything using logic or symbolically, just substitute values of x. After finding the critical points, turning it into domain is as simple as just substituting points and see if the inequality holds.

Domain Calculation

Started the domain calculation file (domain.py) and made the function get_constraints() that recurses through the tree to look for domain restricting operators like /, sqrt, log, arcsin, etc. into a list. next steps are to solve each of the generated constraints (inequalities) and then combine them to get the domain for x (or any variable).

Currently working on trying to solve the inequalities but it is proving to be pretty difficult so it could take longer than usual.

AST Visualizer (continued)

Case-2

If the node is not a leaf (i.e. not a string) we will again check for 2 cases. the node can either have 1 child or 2 children. If there are 2 children then we have to make 2 recursive calls but if we have a single child then only one and the other stuff is pretty similar so I will only explain the case when there are 2 children. first of all, we unpack the node to get op, left and right. Now we call assign_positions with depth+1 (to effectively track each level we go down), on left and right and extract: x-position of children, id of children and the parameters that get updated (id_count and leaf_count). next, we used children’s x-position, average it to get the op’s x-position. We make the tree: tree[op_id] = {'label': op, 'x': op_x, 'depth': depth, 'children': [l_id, r_id]}. Now we increase the id_count by 1 but not the leaf_count as we have not encountered a leaf (I had made a mistake here and increased leaf_count in the case where the node is not a leaf because I had named the variable as x_position earlier and thought that it must increment everywhere but forgot that we are averaging the children’s x-position to get op’s x-position so i do not need to increment it, which is why I have cleared this up). Now we return the same stuff: op_id, id_count, op_x, leaf_count, tree.

Rendering Tree

The actual tree generation is simply looping through the dictionary. The important function is converting x_position and depth values that are simple integers into actual pixel positions to actually position the circles in the correct spot which is done by pixel_x = margin+x*x_spacing and pixel_y = margin+depth*y_spacing where margin, x_spacing and y_spacing are all hyper-parameters. The next function is draw_nodes which simply loops though the values of tree dictionary gets the x and depth for each node, converts them to actual coordinated using get_pixel_coords and then creates the svg line:
'<circle cx="{x}" cy="{y}" r="25" fill="white" stroke="black" /> <text x="{x}" y="{y}" text-anchor="middle" dominant-baseline="middle">{i['label']}</text>' for each node and its text. The other is draw_lines. It takes all the parent nodes and gets their x and y value as pixel coordinates. Then, it gets the children of the node and for each child’s id, the function gets the child by its id, then gets the x and y coordinates of the child as pixel coordinates and then draws the line as per this syntax:
f'<line x1="{x1}" y1="{y1}" x2="{x2}" y2="{y2}" stroke="white" />'
Finally we have the render_expression function which wraps everything into a single sequence of operations and creates the final svg code:
f'<svg xmlns="http://www.w3.org/2000/svg" width="850" height="650"> {lines} {nodes} </svg>'
and created an svg file to view the final tree.