ShiftGuard AI
- 7 Devlogs
- 18 Total hours
An machine learning research project that detects when neural-network predictions become unreliable under distribution shift.
Finished the first full distribution-shift experiment using Gaussian noise across six predefined severity levels. The baseline CNN’s accuracy dropped from 72.24% to 24.72% at the strongest noise level, while mean confidence only decreased from 78.12% to 68.28%. This caused the confidence-accuracy gap to grow from 5.88 to 43.56 percentage points, showing that the model becomes much less reliable without becoming proportionally less confident. I also added automated CSV logging and generated plots, like the one attached, to visualize the trend.
Implemented the first controlled distribution shift for ShiftGuard: Gaussian noise with six predefined severity levels from σ = 0.00 to 0.25. The transformation adds noise before normalization, keeps pixel values within [0, 1] and preserves the original CIFAR-10 image shape. I also added input validation and a reproducible smoke test, which passed successfully. Next, I’ll evaluate the trained baseline CNN across all noise severities and compare accuracy with prediction confidence.
Evaluated the trained baseline CNN on all 10,000 unseen CIFAR-10 test images. The model reached 72.24% test accuracy with 78.12% mean confidence. Correct predictions had 84.34% average confidence, while incorrect predictions still had 61.93%, which gives me the first clean baseline for investigating overconfidence under distribution shift. Next, I’ll introduce controlled image shifts and compare how accuracy and confidence change.
Trained the first ShiftGuard baseline CNN on all 50,000 CIFAR-10 training images using my AMD GPU. Over 5 epochs, training accuracy increased from 53.47% to 80.57%, while the loss dropped from 1.3057 to 0.5520. I used the optimizer Adam with a learning rate of 0.001 and normalized the CIFAR-10 images before training. The trained weights are now saved, so the next step is evaluating the model on the unseen test set and measuring its prediction confidence.
Built and tested the first baseline CNN for CIFAR-10 and verified that the model correctly outputs 10 logits per image.
Loaded and visualized CIFAR-10 data (16 random training images) and prepared the dataset pipeline for the first CNN experiments.
Set up the ShiftGuard AI repository, Python virtual environment, ML dependencies, project structure and environment test. Defined the initial research question.
