Final Devlog: Finished everything, wrote readme, added website (terroranalyzer.streamlit.app), tried regularization (to no avail).

Update: Linear Regression Almost Over!

I did three things over the past few hours:

• Found out if Inequality affects Terrorism
• Added lag features (making the model autoregressive), allowing me to predict terrorism based on past terrorism
• Actually built a linear regression model on my data, and assessed it.

First things first: Inequality. Similar to my process in the last devlog, I added another dataset (Ethnic Power Relations Dataset), cleaned it, created a metric to measure power disparities between groups, and graphed it. I couldn’t see a relationship, but there is a lot of room for improvement here. Didn’t change anything because there was no time left.

Second: Autoregressive Nature. I had several years worth of data, which meant I could use past terrorism to predict future terrorism. Environmental factors are insufficient alone, as they predispose not guarantee (similar to genetics). Plus, I was only able to find one factor that correlated with terrorism (perhaps I should’ve read some papers to figure out what correlates in reality. My mistake for not doing so.) Hence, to avoid creating a completely unreliable model, I aggregated my data, and created 2 columns - delay1 and delay2. Delay1 is essentially data of terrorism from 1 year back, and same for delay2.

Third: Linear Regression Model: Through plotting, I found out all my data shared some sort of power law relationship (probably because a few attacks kill many while many smaller attacks kill none). This made my job easier, since I could just put everything through a log. I did so, and I accidentally put year there too. As this model is autoregressive, and I want to avoid assessing the model on data it’s already seen, I had to implement a year cutoff (years 2013 onwards in the test set, the others in the train set) instead of a train test split. I used scikit-learn to fit the model to the data.

Assesment:
First assessment shows the model’s alright. It’s not great at predicting outliers but it’s median absolute deviation is acceptable. I had to exponentiate the results, in order to reverse the logarithm that I’d applied previously. A caveat - it’s terrible at predicting outliers. I don’t have too much time to modify anything else, maybe I’ll add some sort of regularization. It’s my first linear regression model, and I’m happy it works.

PS: autoregressive means the model uses previous data to predict future data. Theoretically, it could be trained on its own outputs.

###Devlog 5

You might wonder why random red dots are flying to heaven.

Short story: It’s one of my most annoying failures.
You see, I’m building TerrorPredictor, a linear regression model that predicts terror attacks in different countries.

Yes, this is AI but it’s old school.

Anyway, over the last few devlogs (go check them out!) I’ve collected data to feed to the model (so that it can learn). You see, terror attacks are driven by many things including the living conditions of the population. Or so I thought. This graph shows me that’s not true, at least in my data.

———–Long Story————

Over the last few days I’ve been learning statistics. I’ve mostly spent time on theory, which is why only 3 hours of work is shown here. I’ve learnt a lot, and made some mistakes too.

First of all, I learnt about variance, standard deviation and other metrics that we use to study data. It turns out my data is not uniform, because some terrorist attacks have thousands of victims, while most kill nobody. This spelled trouble for my analysis, because the most common analysis method, linear regression, requires the data to be more or less uniformly distributed. But this is an issue which can be fixed.

——What’s the graph?——

I’ll answer that soon. After looking at metrics, I started looking at statistical distributions. I plotted my data, and found this graph. At first glance, it looks like there is some sort of relationship between HDI and number of victims of attacks. Notice the floor of values near y=0. It tells me that it’ll be very difficult to predict anything using HDI. I’m not getting rid of HDI data yet, though things don’t look good for it. (HDI is the human development index - a metric quantifying living conditions)

I learnt a painful lesson though - before spending effort cleaning data (other than the bare minimum of course), first check that the data has a relationship at all.

——What Works?——

I did have some success - civilian deaths in a country seem to have a power-law relationship with number of casualties, so I do have one successful parameter. I can easily convert this to linear through log-log scaling.

I’ll try to look for more parameters - probably already found one too. I won’t make the same mistake again.

PS: The reason the dots are reaching for heaven is because they are outliers and don’t follow the typical pattern.

Finished Data Cleaning

Devlog 4
Previously, I’d cleaned the Global Terrorism Dataset which gave me information of past instances of terrorism in a country. I’ve merged two new datasets in, the Uppsala University Georeferenced Events Dataset, from which I calculated the civilian death toll of a conflict, and the UN’s Human Development Index, which I used to asses the living standard of the country. I hope these metrics will be sufficient to give me an accurate analysis.

My learning:
The right dataset matters more than how you clean it
Perhaps this isn’t true for big data, but I’ve found that the right dataset can save like half the cleaning. I tried using ACLED, the One Sided events database from Uppsala, and looked at several other metrics (I originally planned to use GNI for economic situation, but I found HDI is a more accurate metric). The UN dataset was wonderfully maintained, and I didn’t have to clean it too much. The merge did take some trial and error, with column renaming and all that, but overall the work was easy.

About TerrorPredictor:
It’s an ML model that uses publically available data about factors relating to conflict, in order to the number of terrorist attacks for a given country in a year.

Over the last few hrs I’ve been finishing the data cleaning of global terrorism dataset off. I’ve imputed what I need and cleaned the dataset sufficiently. The next step is to look for data of the environmental factors, namely:

• Economic Situation
• Brutality of Government forces
• Crime rate and other related stats
  After I get this information and clean the data, I should be able to build the predictor.

I had been able to get in a flow state last time, so I had reasonably good quality of work. I had a few issues and a few learnings too.

• I really struggled with using groupby to get analysis; Essentially what I wanted was all terrorist groups arranged, by country, in descending order by kills , i.e most kills first, least kills last, by country. To create this, I had mainly tried to use groupby() in different ways, however my solution ended up using both groupby, and sort, and set index. I learnt that the most obvious solution isn’t always the right one.
• I also learnt the utility of lambda functions - they’re really useful for filtration and other stuff that normal functions won’t suffice for.
• Additionally, I learnt about a technique to make method chaining cleaner, by using () and writing methods inside that, line by line. An example is given in the picture.
  Overall, lot of work and significant progress. Cleaned code is available on github.

Global Terrorism Dataset - Cleaning
Over the past few hours I’ve been cleaning the global terrorism dataset. So far, I’ve removed columns that I do not require (over half the dataset), parsed dates, and cleaned the group name column (which is important to identify organizations). You can find my entire pipeline on github, but here are some places where I faced issues and this is what I learnt.

1. AI wiped out my entire code
   I’m not using AI to code for this project, other than to look up functions for a specific usecase (i.e no coding agent). One place I’ve extensively used AI is Version Control in git, to help me remember commands. Long story short, I had uploaded the entire dataset to github and needed to remove it because the file size was too much. I used an AI generated git command and it wiped out my entire codebase. Thankfully I had a window open so I didn’t lose my code.

My learnings:

• Version control is only for code, not files.
• NEVER trust AI with mission critical work. Always know what it’s doing, even in mundane work like version control.

1. I almost wasted hours typing
   The global terrorism dataset contains many columns, like attacktype1, attacktype2, target1, target2, and so on. Most of these are of no use. I was originally gonna put all of these in a list and remove them, and given that there are over a hundred columns, I could’ve wasted like an hour typing. Thankfully, I realized that this had to be automatable, so I learnt about the any() function and wrote a nice list comprehension to generate that list of columns to remove for me. AI did help here, and I realized that was a problem.

My learnings:

• Advanced Python, like list comprehensions, is very helpful while cleaning data.
• Avoid repetition at all costs
• An imperfect solution should be made before using AI for support. It leads to learning, and the AI can act as a tutor to help make your pipeline more efficient.

the attached pic is a cleaner dataset, btw

How I implemented these learnings
While parsing dates, I created an imperfect solution first. The only use of AI was to find the difference between astype() and to_string() functions. The logic for date parsing was entirely my own, done by combining idate, imonth, and iyear columns. This actually allowed me to learn.

Welcome to TerrorPredictor! It’s a data analysis and Classical ML project which will use publically available data on terrorism, such as the Global Terrorism Database (that’s the only one I started with, will add more data soon!) to predict the number of terrorist attacks in a given region or country. The only problem, of course, is that I have no skills to build this thing! That’s what I’ve been doing for the past few hours (it’s more than five hours in tutorial hell).

Main Components
This refers to the components of code on the backend. Firstly, I’ll need the data itself, which I plan to take from Kaggle’s Global Terrorism Database and some other sources. Some of the factors I’m thinking of considering are:

1. Economic situation
2. Local Culture
3. Political Stability of the Area
4. Conduct of government forces and their allies
   .. and a whole lot more.
   To be honest, I don’t really know how many factors will be necessary. I fully expect this project to fail miserably. But it’s worth a try.
   To build this thing, I’ll need to know:
5. Pandas for data cleaning and basic insights
6. Classic ML algorithms such as Linear regression, and maybe some tree based models
7. Numpy, to process the numbers
8. And of course, plotting libraries to actually understand the data.

The past few hours I’ve spent on Pandas. It’s vast and boring, but that’s my fault for over relying on tutorials. I’m changing track today, and I’ll be cleaning a part of the Global Terrorism Database, to ground my learnings.
Thanks for reading this devlog!