Devlog #4: Card Detection!!!!!

After spending a painful 8 hours yesterday trying to make an OpenCV-based card detection pipeline work (which I, of course, forgot to track 😭), I finally accepted that it just wasn’t the right approach. Instead, I switched over to object detection using YOLOv8s, a relatively small model with only about 11.2 million parameters.

To start, I had to build a dataset. Today I spent around 4 hours taking pictures of my cards under different lighting conditions (again didn’t track), backgrounds, angles, and distances. By the end, I had collected roughly 500 images, and then came the fun part: manually labeling every card with bounding boxes.

Once I had the initial dataset, I started generating synthetic training images. Using OpenCV, I created templates of each card and placed them onto randomly selected backgrounds that I downloaded online. I also applied random perspective warps, brightness changes, rotations, and other augmentations (an example is shown in the third image).

This let me expand the dataset to around 10,000 automatically labeled images, giving the YOLO model much more variety to train on.

I trained the network for 150 epochs using roughly 8,000 images for training and held out about 2,000 images for validation. Even with a pretty decent GPU, training still took around 6.8 hours.

The payoff was definitely worth it. The model reached about 98% validation accuracy on images it had never seen before. You can also see this in Image 2, the confusion matrix, which is almost perfectly diagonal, meaning the model rarely confuses one card for another.

Since the rest of the project is being written in Rust, I also exported the trained model to the ONNX format so it can be run directly using ONNX Runtime. That should make integrating the detector into the rest of the system much cleaner.

The next step is getting a live webcam feed from the BeagleBone Green to my computer, running inference with the YOLO model in real time, and sending the detected cards back to the Blackjack advisor. The main issue is that the model detects both corners of the card if visible, so we need to merge that into one card somehow. Also detecting your hand vs. the dealer’s hand vs. other players’ hands is another problem I will need to solve.

Current Project State

I will release the best model’s weights in GitLab releases soon.