@Matrix

I was studying the outputs produced by my system across various channels , and was plotting multiple graphs , comparing multiple parameters to understand it .And i found out what i need to do next to increase my model efficiency , i know i cant make it perfect because i am currently working on a real dataset by NASA , also i dont have enough time because of my exams . so my next plan is to add an intelligence layer . Like before i was hardcoding the the thresholds like 2std or 3std and the min_consecutive value logic . so i decided to let my system decide it what is best for it . like i am gonna first train my lstm model on validation set and than like compute threshold with the validation error and also the min_consecutive logic and find the best pair and than test it with the testing dataset . and in past hours i also tried to add the option for user to select the model himself to see the output and result , implemented the random forest one , linear regression is left i think i will do it after the lstm improvement .so stay tuned

Update – Like i made an basic dashboard for my system .Like i separated it in 2 parts , first my previous stimulator based on my maths concept where i myselft created the random anomlies by inputing the no of request and prob of failures and than used z_Score or poission distirbution accorfing to which conditon it satisdy to conclude the results . Thats was V1 of my project , than added bayes concept like where my system pretend it doesnt know the cause , and it finf the p(cause|evidence) which is finding the prob of cause given evidence , that was v2 . now for v3 i created an different system where i added the nasa dataset and than run the models . So i put both of those to show my actual progress , and like user can play with both ones . now i think i will polish it more and than focus on improving the accuracy of lstm .

I implemented lstm , but its not that accurate , it works great and precision results are awesome , some channel precision were even 100 , but recall is mostly
average i need to improve it like its 26 - 45 % . But heres the problem , my school exams are coming and i need to start prepare for that urgently and cherry on top i have this weekend where i can try to give more hours but next week schedule is gonna be tough but i will still try to code and debug and plan everything and solve the problem . so i have decided first to make a dashboard using streamlit in python . That will include V1+V2 and V3 section where user can choose input like channels from NASA dataset and see the results and found anomalies in the system .So i am working on that and postponding my schedule to improve lstm later , because if i try to improve lstm now i dont know how much time it can take and like time will left for making dashbaord so i need to do this first because we get voting on usability section also and i dont it to be cli .

Update – i completed my analysis with linear regression and random forest model . I calculated the perdect tuning by changing window size to 10 and found out the best threshold for anomalies , my formula for threshold was x * std + mean , and chosing 0.5 as x gave us the best result and it makes as i realize because prediction errors during anomaly range arent that much bigger compare to errors in normal range . so thats why lowering down the x helped us . Also i created confusion matrix and get result for random forest and linear regression model, catching around 42 percent anomaly though only 12-13 % catched were actually correct . Tested the setup on multiple channels , some channels work perfectly and give some good results but some not and with no cleat pattern i end my case with lienar regression model and random forest here and moved towards implementing LSTM . Currently implemeting it and testing it and trying to improve the results . LSTM update will come in next devlog .

Update , like i finshed anlayzing the sensors . I plotted multiple sensors on graph and like for point i can observe pattern in the graph for changes like high spike in anomaly zone , but for contextual there was no similarity or pattern . even for point the spikes are not similar for every sensors . Some magintude and std is more compare to other . So i conclude that we cant make an rule based system for this dataset . I have to move towards implementing ml . So i used linear regression model first to train on first 10 input and than predict the 11th ouput , as i am doing time series analysis because my dataset is continuous data , so i have kept my window size 10 . so i am currently analyzing how many errors lie in our anoamly zone . If linear regression model didnt perfectly work i will try differnet model like random forest and if they also didnt work i have to move towards lstm .

I added real Nasa dataset to my system . Its NASA Anomaly Detection Dataset SMAP MSL . Its vast dataset have different sensors sendsing the telemary and giving the values i have been studying the data from the past 4 hours like across different channels to observe pattern . i currently found that channels of point like p-1 , e-11 have different pattern compared to contextual . like i calculated the changes in their consecutive values , and check the values under anomaly range provided inside the dataset . point sensors have very large amount of change inside their anomaly range and contextual have long runs of zero inside their anomaly range . I am still studying the data to conclude more information and patterns .

After building Bayes, I added a 3x3 confusion matrix to check its accuracy , it tracks how many predictions were correct versus which causes got mixed up with each other. Then I ran the simulation 10,000 times and saved the accuracy from each run into a list, and the average accuracy came out to about 98% . Next step is adding the NASA dataset to make it an real stimulation system .

So after hours of work i successfully added Bayes logic , it was not an easy task because to find p(cause|evidence) i needed to find p(evidence|cause) and p(cause) and p(evidence) for each possible cause . And i was so confused because i used lots of loops to implement this , and after hours i finally did it . And now for every given evidence i have the most probable cause . The bayes is gonna be more useful in future when i will add real dataset from Nasa and ml concepts also .

Been learning probability and statistics for my ai engineer roadmap for the past month but it was all theory, no projects, so when i found stardance I figured this was a good chance to actually build something with it instead of just reading. The project stimulates a system getting a bunch of requests with some chance of failing, then uses stats to check if the number of failures is normal or if somethings actually wrong, using binomial and poisson to figure out how many failures we’d expect and comparing that to what we actually got. First version used a z-score for this but while testing with small numbers I noticed the results got weird, turned out z-scores don’t work well when the expected failure count is small because the math assumes a symmetric bell curve and at small numbers the real distribution is skewed, so I ended up building an exact fix using the actual poissonon formula instead of the approximation for those cases. After that I started working on figuring out not just that something’s wrong but why, so now every failure gets assigned a hidden cause like database overload, network issues or server overload along with evidence that actually matches it, like high db load for database issues and high latency for network stuff, but with enough overlap that it’s not too obvious or easy. Next up is bayes theorem, where the program only sees the evidence and has to figure out the probability of each cause itself, which is when it goes from just flagging that something’s off to actually explaining why.