By
@s_ray9
independent research project investigating feature reliability in machine learning prediction of cognitive impairment from sleep-derived physiological signals.
I built a research pipeline to evaluate whether ML model feature importance is trustworthy given a small, imbalanced medical dataset. What’s exciting is that it uses the relatively new 2026 PhysioNet Challenge sleep dataset, so it’s new stuff!
It was a huge pain getting the pipeline to work reliably. I had to reorganize the data, which kept messing stuff up, and it was pretty slow and uneventful at times. I had to deal with file mismatches, adapt to missing files, and figure out how to handle licensing between my code and the original Challenge code. However, the real challenge was validating results and catching a mistake along the way. Initially, I thought a feature like arousal frequency was the strongest predictor after two separate runs. At the time, it seemed promising, but I was unsure. When I tested it across 20 random seeds, it completely died. I’m pretty glad I kept digging deeper and running precautionary tests; I’m proud of realizing that mistake, which is a crucial skill in real research.
Anyways, if you want to try it, there’s no clickable button. Since it’s analysis code, you’ll have to clone the repo, follow the setup in the README, and run the scripts yourself. You should get the same numbers I did. If not, please let me know. Oh, and please let me know what you think, or if I made a mistake (you can never be too careful)!
17m 2s logged
I just deployed the core project documentation and README to GitHub Pages using Jekyll!
I just standardized the project layout and set up a Cayman Jekyll theme. Some parts of the README kept breaking, so I had to migrate to semantic HTML. Everything is close to finished, if not already.
Go check it out!
6h 47m logged
I just reached my first major milestone in building a full ML pipeline on the 2026 PhysioNet Challenge sleep dataset! Since it’s a relatively new dataset, I decided to train a model and predict cognitive impairment from overnight sleep recordings.
It took several hours to finish up the presentation and organize the key results. Initially, the baseline model showed promise (92% accuracy). However, it was basically guessing “no impairment” and hoping for the best, since there was only around 8% prevalence of cognitive impairment across all patients.
When I fixed the class imbalance and model setup, the real score was 0.80.What’s cool is that arousal frequency looked like the top predictor across two separate runs, and I thought it was a real finding, but when I tested it across 20 random seeds, it completely fell apart.
Of the full 75 features, the results indicate that only age and BMI really mattered, even though the model performed best after only seeing the top 10 features. Interestingly, those extra, noisy features actually did something: they degraded the model’s performance.
