Ship #1 Pending review

@Vedant_Daga on Machine Learning in Medicine · 8 days ago

## what i made
a system that tests whether ai can actually detect disease from your voice, honestly. turns out no for a cold (0.54 auc, doesn't generalize), but yes for parkinson's (0.83) and voice disorders (0.86), once you split patients properly and test on data the model's never seen

## what was challenging
catching my own model cheating. it hit 84% on "sick or not" and looked great until i trained a model on zero audio, just age/gender/date, and it matched the same accuracy. had to prove it wasn't real by testing cross-dataset, which dropped it to a coin flip.

## what i'm proud of
not deleting the failures. the repo shows the cold-detection model failing right next to the parkinson's model working, with receipts for both, instead of just posting the good number.

## what to test
grab the trained model from `parkinsons-success/` or `svd-success/`, run `predict.py` on a feature file or audio clip, and check it against the numbers in the readme. everything's reproducible from the pipeline scripts, no hidden steps.

## the gap, in one chart
every experiment's first number vs the honest number, side by side. red ones never recovered, teal ones held up once evaluated properly. the other chart is just the model accuracies and AUC

1 devlog
10h

Try project → See source code →

Open comments for this post

@Vedant_Daga on Machine Learning in Medicine · 8 days ago

9h 54m 55s logged

devlog 1

just built + broke + fixed a voice-disease detector

did this whole thing in one session, not mutliple so this is gonna be one really long devlog so im sorry for that. oopsssssss

what changed:

my model went from “84% accuracy predicting if someone’s sick from their voice” to realizing that number was basically fake (this itself took 3 hours to build and it was kinda sad when i realized it all didnt work)
decided to scrap it and i made everything for two other diseases where the accuracy was actually real (0.83 and 0.86 AUC)

what i built

so this is the first model its a group of models like random forest, svm, etc
a metadata-only model (just age, gender, recording date, zero audio) as a sanity check
a cross-dataset test, train on one dataset, evaluate on a totally different one it’s never seen (this is SUPPOSED to be the standard but liek most medical companies dont so they can inflate their numbers)
had a goodtrain/test splits for the datasets
a decoder for an basically ancient audio thing
one clean pipeline for the liek 100 test runs plus trained models that anyone can download and run

what broke

the 84% “sick or not” model turned out to not be hearing illness at all, it matched a model trained on metadata not acutally what it was supposed to be
cross-dataset test dropped auc to 0.54, a literal coin flip, and nothing i tried fixed it (relabeling, per-dataset standardization, synthetic oversampling) – first parkinson’s dataset hit ~95% which was immediately suspicious, turned out patients had multiple recordings and my split let the same voice leak into both train and test

what i learned

a metadata-only “no audio” baseline is the single fastest way to catch a model that’s cheating
patient-level splitting matters way more than recording-level splits

a decision i made

instead of scrapping the project after the 0.54 collapse, i split the question into two: “can voice reveal illness at all” vs “can voice reveal a cold specifically.” tested the first idea against parkinson’s and voice disorders instead of respiratory infections, since those change your vocal cords/motor control permanently instead of for a few days

the repo, organized

kept the failed experiments in the repo right next to the ones that worked instead of deleting them, felt more honest that way
color-coded results red (didn’t generalize) vs teal (actually held up) so it’s obvious at a glance which is which