Previously, 15 other incorrect cells during training were drowning out the 1 correct cell. I decided to boost the correct cell’s weight on training by doubling it. Then, I increased learning rate to 0.01 and epochs to 25. It’s still training, but difference is night and day.

Epoch: 1
loss 102.3050 | conf@object 0.580 | conf@empty 0.296 | cell 37/100 (within 1: 87) | class 9/100 (chance 3%) | p(true class) 0.382 | p(top class) 0.532
Epoch: 2
loss 94.2365 | conf@object 0.619 | conf@empty 0.257 | cell 52/100 (within 1: 99) | class 13/100 (chance 3%) | p(true class) 0.414 | p(top class) 0.587
Epoch: 3
loss 86.3697 | conf@object 0.699 | conf@empty 0.223 | cell 58/100 (within 1: 98) | class 12/100 (chance 3%) | p(true class) 0.446 | p(top class) 0.644
Epoch: 4
loss 84.6920 | conf@object 0.752 | conf@empty 0.189 | cell 60/100 (within 1: 100) | class 16/100 (chance 3%) | p(true class) 0.496 | p(top class) 0.654
Epoch: 5
loss 85.4948 | conf@object 0.796 | conf@empty 0.166 | cell 61/100 (within 1: 100) | class 24/100 (chance 3%) | p(true class) 0.532 | p(top class) 0.676

Previously, the number of times the network got the correct object in the cell was 3-9%. Now, in only the 5th epoch (out of 25), it is up to 24. There is still a long way to go, but I found this change significant.