I built AeroVision this spring as a computer vision tool that classifies structural damage in aircraft components: things like cracks, corrosion, dents, and scratches using a deep learning model (EfficientNetB0). The idea came from reading about how aircraft inspections are still mostly done manually by technicians visually scanning for defects, which is slow and prone to human error bc they are under time constraints. I wanted to see if a trained CNN could automate that classification step.
I trained the model on a public dataset of 6 damage classes and built a Streamlit app around it with saliency maps (visual heatmaps showing exactly where the model was “looking” when it made a decision), a PDF report generator, and a dual-model reliability system that cross-checks predictions between two models before flagging high confidence in the model’s decision. I submitted it to the Virginia Invention Convention and ended up as a state finalist and won a Databook award, which was a big deal for me.
But after VIC I went back and looked at the actual test accuracy numbers more carefully. That’s when I found the real problem. After VIC I reran my model evaluation properly and found something I had missed earlier: my test accuracy was only 63%, even though my training accuracy was 98%. That 35 percent diff was actually overfitting, which is when the model memorized my training images instead of actually learning what the different types of damage looks like.
When I dug into why, the answer was in my class distribution. My Crack class had 1,269 training images. My Corrosion class had 51. That’s a 25x imbalance. The model had basically never seen enough corrosion examples to learn what corrosion actually looks like, so it was defaulting to guessing Crack or Paint Off whenever it was unsure. Every single misclassified example in my confusion matrix was Corrosion being predicted as something else.