MIRA — Day 4: Evaluate and important results

What I Built

Today I built evaluate.py and redid all of my images so my data and captured 800 increasing the count from 120 to 800. This allows me to load my 800-image dataset, evaluate it against a validation set, and output a visual Confusion Matrix.

Vital Changes & Difficulties Faced

1. The Alphabet-Slice Bug (Deterministic Split vs. Batch Shuffling)

When I first ran evaluate.py, my Confusion Matrix looked bizarre: I had 159 test files, but every single one of them had a true label of “plastic”!

To keep my evaluation predictions aligned with the correct labels, I had initially set shuffle=False when loading the directory. Because Keras loads folders alphabetically (glass -> metal -> paper -> plastic), disabling shuffling caused it to read the sorted list and simply slice the last 20% for testing. The last 20% of the alphabet was entirely plastic.

To fix this, I set shuffle=True so Keras would pull the exact same random validation split as the training script. To prevent the predictions from getting mismatched during shuffling, I refactored the evaluation script to pull images and labels simultaneously, batch-by-batch:

y_true = []
y_pred = []

for images, labels in val_ds:
    preds = model.predict(images, verbose=0)
    batch_preds = np.argmax(preds, axis=1)
    
    y_pred.extend(batch_preds)
    y_true.extend(labels.numpy())

No matter how randomly Keras shuffles the images, our answers and guesses are extracted at the exact same instant, making mismatching impossible.

The Paper vs. Metal Crisis 📊

With the pipeline corrected, I ran the evaluation and got a real, stable baseline accuracy of 61% worse than yestarday which was expected as the images were alot more than before and before them it was just learning the images and not guessing. Across my 159 validation images. The Confusion Matrix revealed a major failure point in my baseline CNN:

• True Paper Images: 42
• Correctly classified as Paper: 3
• Misclassified as Metal: 25 (59.5%)

My simple 3-layer CNN is nearly blind to paper textures. It looks at white paper or cardboard with bright reflections and thinks: “Ah, a bright shiny reflection! This must be a metal can.”

What’s Next

Now that I have documented my true baseline and identified its major blind spot, we are ready to implement Transfer Learning (MobileNetV2). This pre-trained model has seen millions of images and knows how to distinguish between the flat texture of paper/cardboard and the metallic shine of aluminum. Also i noticed that I struggled quite alot with this part so iam going to pause this project a bit to do another project ot solidify my python basics and continue from there onwards.

Resources & Documentation

• Keras Sequential Model API — Documentation used to structure layers, dropout, and data augmentation.
• Scikit-Learn Classification Report — Reference for evaluating precision and recall.
• Scikit-Learn Confusion Matrix Display — Documentation used to plot the final matrix.