okay so this one’s been in my head for a while. every time I’m debugging something slow on linux I end up with like 3 terminals open — strace in one, perf in another, sometimes py-spy if it’s python — and I’m just manually lining up timestamps trying to figure out what actually caused the slowdown. none of these tools talk to each other. they just dump their own little slice of the truth and leave you to do the detective work.
so I built whyslow. it hooks into the kernel with eBPF and watches three things at once: scheduler switches, futex waits, and block I/O. instead of showing you three separate logs, it actually chains them together and tells you the story.
like instead of
thread stuck for 400ms (somewhere)
you get
thread A blocked on a futex
which was held by thread B
which was blocked on a disk read
that’s the whole point. not “something was slow” but “here’s the chain of custody for why.”
biggest pain building this was honestly not the eBPF part, it was figuring out how to correlate events that come from completely different sources without just guessing. had to build this whole interval-based matching thing to line up “thread went to sleep here” with “thread got woken up there” and make sure the causality actually holds up and isn’t just coincidence in timing.
kept scope small on purpose for v1. only 3 event types, no fancy stack traces yet, x86_64 + arm64 only. figured getting 3 things exactly right beats having 8 things that are kind of right and kind of not.
tested it by writing a little program that deliberately deadlocks itself with a futex and forces a disk read, so I’d have a known answer to check whyslow’s output against. felt really good when the chain it printed out actually matched what I knew was happening under the hood.
still needs root to run since eBPF just doesn’t let you do this stuff unprivileged, that’s a permanent tradeoff not a bug.
next up is probably symbol resolution so it can tell you actual function names instead of just thread ids, and getting it properly published so people can just cargo install it instead of pulling from git.
repo’s here if anyone wants to poke at it: github.com/kaorii-ako/whyslow