Been learning probability and statistics for my ai engineer roadmap for the past month but it was all theory, no projects, so when i found stardance I figured this was a good chance to actually build something with it instead of just reading. The project stimulates a system getting a bunch of requests with some chance of failing, then uses stats to check if the number of failures is normal or if somethings actually wrong, using binomial and poisson to figure out how many failures we’d expect and comparing that to what we actually got. First version used a z-score for this but while testing with small numbers I noticed the results got weird, turned out z-scores don’t work well when the expected failure count is small because the math assumes a symmetric bell curve and at small numbers the real distribution is skewed, so I ended up building an exact fix using the actual poissonon formula instead of the approximation for those cases. After that I started working on figuring out not just that something’s wrong but why, so now every failure gets assigned a hidden cause like database overload, network issues or server overload along with evidence that actually matches it, like high db load for database issues and high latency for network stuff, but with enough overlap that it’s not too obvious or easy. Next up is bayes theorem, where the program only sees the evidence and has to figure out the probability of each cause itself, which is when it goes from just flagging that something’s off to actually explaining why.
Comments 0
No comments yet. Be the first!
Sign in to join the conversation.