Tokenizer progress:

1. Added decimal point support:

• Failed attempt: I initially tried to detect . from inside the .isdigit if-elif loop like detecting a continuation of digits but failed on case "12.34" → ['1234'] and then found the simpler solution.
   
• Added elif exp[i] == ".": num.append(exp[i]) as its own top-level case, letting a decimal point unconditionally join whatever number is currently being built inside the num list.
   
• all test cases passed: "0.3 + 1.5436* 245" → ['0.3', '+', '1.5436', '*', '245']. And, "3.5", ".5", "12.34" all correctly maintained their decimal points.
   
• tested all of the previous test cases like multi-digit numbers, multi-letter function names, parentheses, mixed expressions and all passed.
   

2. Added Unary minus v/s binary subtraction distinction:

• While thinking of some more edge cases, I found a case "2*-3" → ['2', '*', '-', '3'] which was wrong and I found out that the tokenizer needs to make the distinction between a unary minus -3, -x, etc. and a binary minus 2-3, x-4, etc.
   
• Added a special condition for - in the top level operator detecting if block and handling edge cases in the .isdigit and .isalpha elif blocks.
   
• All test cases passed:-
  • "3-2" → ['3', '-', '2']
  • "3*-2" → ['3', '*', '-2']
  • "-2+3" → ['-2', '+', '3']
  • "(-2)" → ['(', '-2', ')']
  • "3--2" → ['3', '-', '-2']
     

3. Added support for new operators and functions:

• Support for ^ operator by adding it to the top level operator detecting if block.
   
• Support for Absolute Value |x| by adding it to the top level operator detecting if block.
   
• Support for Fractional Part Function {x} by adding it to the top level operator detecting if block.
   
• Support for Greatest Integer Function [x] by adding it to the top level operator detecting if block.
   
• Also added these cases inside the unary minus handling if block thus passing all possible test cases.
   

---

 

Status:

The tokenizer part is complete for now. There are a few changes to be made, like adding support for implicit multiplication. Though, it is something that could be handled by the parser so I will handle it as time comes.