I tried QT-Opt and at least in simulation, it did a lot better (it also trained much faster due to the other architectural change I made; see next). I also switched the backbone to Dinov2 Small, which is newer and smaller than the previous Dino Base backbone.