TerrorPredictor

Over the last few hrs I’ve been finishing the data cleaning of global terrorism dataset off. I’ve imputed what I need and cleaned the dataset sufficiently. The next step is to look for data of the environmental factors, namely:

• Economic Situation
• Brutality of Government forces
• Crime rate and other related stats
  After I get this information and clean the data, I should be able to build the predictor.

I had been able to get in a flow state last time, so I had reasonably good quality of work. I had a few issues and a few learnings too.

• I really struggled with using groupby to get analysis; Essentially what I wanted was all terrorist groups arranged, by country, in descending order by kills , i.e most kills first, least kills last, by country. To create this, I had mainly tried to use groupby() in different ways, however my solution ended up using both groupby, and sort, and set index. I learnt that the most obvious solution isn’t always the right one.
• I also learnt the utility of lambda functions - they’re really useful for filtration and other stuff that normal functions won’t suffice for.
• Additionally, I learnt about a technique to make method chaining cleaner, by using () and writing methods inside that, line by line. An example is given in the picture.
  Overall, lot of work and significant progress. Cleaned code is available on github.

Global Terrorism Dataset - Cleaning
Over the past few hours I’ve been cleaning the global terrorism dataset. So far, I’ve removed columns that I do not require (over half the dataset), parsed dates, and cleaned the group name column (which is important to identify organizations). You can find my entire pipeline on github, but here are some places where I faced issues and this is what I learnt.

1. AI wiped out my entire code
   I’m not using AI to code for this project, other than to look up functions for a specific usecase (i.e no coding agent). One place I’ve extensively used AI is Version Control in git, to help me remember commands. Long story short, I had uploaded the entire dataset to github and needed to remove it because the file size was too much. I used an AI generated git command and it wiped out my entire codebase. Thankfully I had a window open so I didn’t lose my code.

My learnings:

• Version control is only for code, not files.
• NEVER trust AI with mission critical work. Always know what it’s doing, even in mundane work like version control.

1. I almost wasted hours typing
   The global terrorism dataset contains many columns, like attacktype1, attacktype2, target1, target2, and so on. Most of these are of no use. I was originally gonna put all of these in a list and remove them, and given that there are over a hundred columns, I could’ve wasted like an hour typing. Thankfully, I realized that this had to be automatable, so I learnt about the any() function and wrote a nice list comprehension to generate that list of columns to remove for me. AI did help here, and I realized that was a problem.

My learnings:

• Advanced Python, like list comprehensions, is very helpful while cleaning data.
• Avoid repetition at all costs
• An imperfect solution should be made before using AI for support. It leads to learning, and the AI can act as a tutor to help make your pipeline more efficient.

the attached pic is a cleaner dataset, btw

How I implemented these learnings
While parsing dates, I created an imperfect solution first. The only use of AI was to find the difference between astype() and to_string() functions. The logic for date parsing was entirely my own, done by combining idate, imonth, and iyear columns. This actually allowed me to learn.

Welcome to TerrorPredictor! It’s a data analysis and Classical ML project which will use publically available data on terrorism, such as the Global Terrorism Database (that’s the only one I started with, will add more data soon!) to predict the number of terrorist attacks in a given region or country. The only problem, of course, is that I have no skills to build this thing! That’s what I’ve been doing for the past few hours (it’s more than five hours in tutorial hell).

Main Components
This refers to the components of code on the backend. Firstly, I’ll need the data itself, which I plan to take from Kaggle’s Global Terrorism Database and some other sources. Some of the factors I’m thinking of considering are:

1. Economic situation
2. Local Culture
3. Political Stability of the Area
4. Conduct of government forces and their allies
   .. and a whole lot more.
   To be honest, I don’t really know how many factors will be necessary. I fully expect this project to fail miserably. But it’s worth a try.
   To build this thing, I’ll need to know:
5. Pandas for data cleaning and basic insights
6. Classic ML algorithms such as Linear regression, and maybe some tree based models
7. Numpy, to process the numbers
8. And of course, plotting libraries to actually understand the data.

The past few hours I’ve spent on Pandas. It’s vast and boring, but that’s my fault for over relying on tutorials. I’m changing track today, and I’ll be cleaning a part of the Global Terrorism Database, to ground my learnings.
Thanks for reading this devlog!