S.I.G.N.A.L (sign interpretation & gesture network algorithm)

I am quite literally writing this on a boat. Today I: fixed the prediction error, broke the hand tracking & letter spelling then fixed it, fixed the finish motion being detected while signing a word, added the did u mean x or y button initially with a confidence of 80%+ but then updated it to raw confidence of 95%+, i then fixed the progress bars that I broke. tomorrow I’m going to fix collect_words.py, add 10 words to it, fix spelling accuracy and add a tense button. oh and also i’m going to make this readme a bit more formal so i can present it to my teacher. – this is taken from my readme on github which u cna find linked to this project! the screenshot attached is where collect_words.py fails me

ran out of time so I couldn’t do everything I said I would. I did add a 3 second cooldown to word detecting & made word tracking more attractive, and attempted to add more words into collect_words.py but the algorithm kinda broke and it didn’t save my 30 samples. I committed the code to here so if anyone sees what’s wrong pls!!!! reach out & lemme know (i know no one reads these but its worth a shot). tomorrow: im going to fix collect_words.py, add 10 words to it, fix Prediction error: X has 126 features, but HistGradientBoostingClassifier is expecting 123 features as input, add a did you mean x or y button and finally add a tense button like at the end when the sentence is finished there will be 3 buttons that are like: present, future, past, and the algorithm will restructure the sentence accordingly. the photo is the timeout error i got after finishing my 30 samples for collect_word.py subsection DON’T WANT

Today was a longer update, I fixed my train_py because everything it was detecting was -, got started on my word dataset ( I have 9 words so far), added an alphabet mode to word mode to detector.py, updated detector.py to match the new files, created two new files called collect_words and train_words - I use collect_words to record 30 clips of each sign, and then it adds them to asl_words.csv and I use train_wrods to train the model. I also updated my gitignore to not track the audio cache. Tomorrow: I plan to add roughly 20 words into the dataset, make hand tracking more accurate, and add a cooldown to the word detecting as it just spams words.

Today I worked on a couple of things:

• refined my testing system (basically made it more detailed so i could see which letters were the most frequently confused). as a result, i saw that the most common mix-ups/issues were: n vs a (closed fist gestures), r vs u (crossed & parallel fingers, both 0.89% precision), j (motion sign thats represented statically) & i & j confusion.
• i added the puncutation selection tool where . = neutral ! = angry / happy (there’s a sub-menu that pops up if you choose !), and ? = question / suprised.
• i made it so that when you change the punctuation, the pitch of the voice changes. i had an issue where the pitches i entered weren’t properly in hertz but i went on to fix that.
• next, i made the algorithm stop reading (angry) and (happy) by removing them entirely.

tomorrow:

1. i’m going to eliminate the emotion detector, there’s no need for it anymore.
2. try to fix the main mix-ups / problems (possibly eliminate the static problem? is there a way to make the algorithm detect things dynamically – idk?)
3. FINALLY face my fears and try to introduce basic words.

devlog #1 on here, #12 if we count github. today, i added the option to switch between male & female voices. tomorrow i aim to increase accuracy, & add the option to change punctuation before the voice reads it in the final sentence. more on that tomorrow!