I finished training the network for 25 epochs. It took 2 hrs on pretty old hardware, so I would say it performed decent.
Tracking works almost perfectly. It’s able to distinguish between an object and its background, even if it is not a fruit. I tested this with my finger and it followed pretty good. The network makes errors just picking similar-looking fruits, which makes sense since the training data was not that great (for example blue grapes and blueberries look pretty similar through the eyes of the network).