Global Terrorism Dataset - Cleaning
Over the past few hours I’ve been cleaning the global terrorism dataset. So far, I’ve removed columns that I do not require (over half the dataset), parsed dates, and cleaned the group name column (which is important to identify organizations). You can find my entire pipeline on github, but here are some places where I faced issues and this is what I learnt.

1. AI wiped out my entire code
   I’m not using AI to code for this project, other than to look up functions for a specific usecase (i.e no coding agent). One place I’ve extensively used AI is Version Control in git, to help me remember commands. Long story short, I had uploaded the entire dataset to github and needed to remove it because the file size was too much. I used an AI generated git command and it wiped out my entire codebase. Thankfully I had a window open so I didn’t lose my code.

My learnings:

• Version control is only for code, not files.
• NEVER trust AI with mission critical work. Always know what it’s doing, even in mundane work like version control.

1. I almost wasted hours typing
   The global terrorism dataset contains many columns, like attacktype1, attacktype2, target1, target2, and so on. Most of these are of no use. I was originally gonna put all of these in a list and remove them, and given that there are over a hundred columns, I could’ve wasted like an hour typing. Thankfully, I realized that this had to be automatable, so I learnt about the any() function and wrote a nice list comprehension to generate that list of columns to remove for me. AI did help here, and I realized that was a problem.

My learnings:

• Advanced Python, like list comprehensions, is very helpful while cleaning data.
• Avoid repetition at all costs
• An imperfect solution should be made before using AI for support. It leads to learning, and the AI can act as a tutor to help make your pipeline more efficient.

the attached pic is a cleaner dataset, btw

How I implemented these learnings
While parsing dates, I created an imperfect solution first. The only use of AI was to find the difference between astype() and to_string() functions. The logic for date parsing was entirely my own, done by combining idate, imonth, and iyear columns. This actually allowed me to learn.