devlog 1

just built + broke + fixed a voice-disease detector

did this whole thing in one session, not mutliple so this is gonna be one really long devlog so im sorry for that. oopsssssss

what changed:

• my model went from “84% accuracy predicting if someone’s sick from their voice” to realizing that number was basically fake (this itself took 3 hours to build and it was kinda sad when i realized it all didnt work)
• decided to scrap it and i made everything for two other diseases where the accuracy was actually real (0.83 and 0.86 AUC)

what i built

• so this is the first model its a group of models like random forest, svm, etc
• a metadata-only model (just age, gender, recording date, zero audio) as a sanity check
• a cross-dataset test, train on one dataset, evaluate on a totally different one it’s never seen (this is SUPPOSED to be the standard but liek most medical companies dont so they can inflate their numbers)
• had a goodtrain/test splits for the datasets
• a decoder for an basically ancient audio thing
• one clean pipeline for the liek 100 test runs plus trained models that anyone can download and run

what broke

• the 84% “sick or not” model turned out to not be hearing illness at all, it matched a model trained on metadata not acutally what it was supposed to be
• cross-dataset test dropped auc to 0.54, a literal coin flip, and nothing i tried fixed it (relabeling, per-dataset standardization, synthetic oversampling) – first parkinson’s dataset hit ~95% which was immediately suspicious, turned out patients had multiple recordings and my split let the same voice leak into both train and test

what i learned

• a metadata-only “no audio” baseline is the single fastest way to catch a model that’s cheating
• patient-level splitting matters way more than recording-level splits

a decision i made

• instead of scrapping the project after the 0.54 collapse, i split the question into two: “can voice reveal illness at all” vs “can voice reveal a cold specifically.” tested the first idea against parkinson’s and voice disorders instead of respiratory infections, since those change your vocal cords/motor control permanently instead of for a few days

the repo, organized

• kept the failed experiments in the repo right next to the ones that worked instead of deleting them, felt more honest that way
• color-coded results red (didn’t generalize) vs teal (actually held up) so it’s obvious at a glance which is which