Update: Linear Regression Almost Over!

I did three things over the past few hours:

• Found out if Inequality affects Terrorism
• Added lag features (making the model autoregressive), allowing me to predict terrorism based on past terrorism
• Actually built a linear regression model on my data, and assessed it.

First things first: Inequality. Similar to my process in the last devlog, I added another dataset (Ethnic Power Relations Dataset), cleaned it, created a metric to measure power disparities between groups, and graphed it. I couldn’t see a relationship, but there is a lot of room for improvement here. Didn’t change anything because there was no time left.

Second: Autoregressive Nature. I had several years worth of data, which meant I could use past terrorism to predict future terrorism. Environmental factors are insufficient alone, as they predispose not guarantee (similar to genetics). Plus, I was only able to find one factor that correlated with terrorism (perhaps I should’ve read some papers to figure out what correlates in reality. My mistake for not doing so.) Hence, to avoid creating a completely unreliable model, I aggregated my data, and created 2 columns - delay1 and delay2. Delay1 is essentially data of terrorism from 1 year back, and same for delay2.

Third: Linear Regression Model: Through plotting, I found out all my data shared some sort of power law relationship (probably because a few attacks kill many while many smaller attacks kill none). This made my job easier, since I could just put everything through a log. I did so, and I accidentally put year there too. As this model is autoregressive, and I want to avoid assessing the model on data it’s already seen, I had to implement a year cutoff (years 2013 onwards in the test set, the others in the train set) instead of a train test split. I used scikit-learn to fit the model to the data.

Assesment:
First assessment shows the model’s alright. It’s not great at predicting outliers but it’s median absolute deviation is acceptable. I had to exponentiate the results, in order to reverse the logarithm that I’d applied previously. A caveat - it’s terrible at predicting outliers. I don’t have too much time to modify anything else, maybe I’ll add some sort of regularization. It’s my first linear regression model, and I’m happy it works.

PS: autoregressive means the model uses previous data to predict future data. Theoretically, it could be trained on its own outputs.