An IoT-powered blackjack analysis device that tracks the state of a game in real time and uses brute-force simulation and probability-based evaluation to calculate the mathematically optimal move for each hand, providing highly accurate decision recommendations.
Whilst I have been trying to get deck detection to work with the YOLO model, I decided to create a little static website so you can test out the Blackjack AI portion.
The website is missing some features currently, which I will add soon. It also cannot leverage some of the compute-in-parallel features of my Rust code due to limitations in WASM. Also please do not look at this website’s code… it is a mess 😭
In the future I plan to also upload the demo video to this site as well as a version of “The Pitboss” which can run in the browser (so you don’t have to buy and setup your own Bealgebone green hardware).
Current Project State
Website
After spending a painful 8 hours yesterday trying to make an OpenCV-based card detection pipeline work (which I, of course, forgot to track 😭), I finally accepted that it just wasn’t the right approach. Instead, I switched over to object detection using YOLOv8s, a relatively small model with only about 11.2 million parameters.
To start, I had to build a dataset. Today I spent around 4 hours taking pictures of my cards under different lighting conditions (again didn’t track), backgrounds, angles, and distances. By the end, I had collected roughly 500 images, and then came the fun part: manually labeling every card with bounding boxes.
Once I had the initial dataset, I started generating synthetic training images. Using OpenCV, I created templates of each card and placed them onto randomly selected backgrounds that I downloaded online. I also applied random perspective warps, brightness changes, rotations, and other augmentations (an example is shown in the third image).
This let me expand the dataset to around 10,000 automatically labeled images, giving the YOLO model much more variety to train on.
I trained the network for 150 epochs using roughly 8,000 images for training and held out about 2,000 images for validation. Even with a pretty decent GPU, training still took around 6.8 hours.
The payoff was definitely worth it. The model reached about 98% validation accuracy on images it had never seen before. You can also see this in Image 2, the confusion matrix, which is almost perfectly diagonal, meaning the model rarely confuses one card for another.
Since the rest of the project is being written in Rust, I also exported the trained model to the ONNX format so it can be run directly using ONNX Runtime. That should make integrating the detector into the rest of the system much cleaner.
The next step is getting a live webcam feed from the BeagleBone Green to my computer, running inference with the YOLO model in real time, and sending the detected cards back to the Blackjack advisor. The main issue is that the model detects both corners of the card if visible, so we need to merge that into one card somehow. Also detecting your hand vs. the dealer’s hand vs. other players’ hands is another problem I will need to solve.
Current Project State
I will release the best model’s weights in GitLab releases soon.
After a tedious and treacherous 6 hours of coding… the blackjack engine is consistently beating the house. I had to rewrite the code as my initial code was not clean and had tons of bugs.
The engine supports a pretty large ruleset: here
It uses a bruteforce algorithm which explores all of your possible options and then evaluates the expected value of each choice based on the probabilities of other cards showing up for you and the dealer. It is currently capped to depth 8 for the hit action.
Splits are approximated to avoid the exponential increase in computation that comes from recursively simulating every possible sequence of re-splits and draws. Instead, each split hand is evaluated independently by averaging over all possible second cards, calculating the best EV (stand, hit, or double) for that hand, and then doubling the result to represent the two split hands. The error this introduces is tiny and the computational and complexity costs an exact solution would bring outweighs the benefits (marginal).
For determining bet size, the betting module calculates the optimal blackjack bet based on the player’s estimated edge, bankroll, and table limits using Kelly Criterion and configurable betting ramps. It is able to determine when to bet high even better than human card counters as it is able to keep a perfect count (it knows how much of each rank of card is remaining in the deck rather than whether there are more high cards left, which is called Hi-Lo counting, a technique used by humans).
As you can see in the simulation (might have to open the image in a new tab to see all the data), it achieved about a 3.7% edge over the house (NOTE: the ruleset used does not benefit the house as much).
Next steps are wiring this up to the Beaglebone’s OLED display which I created the driver for previously and allowing for the Webcam to connect to the board to autodetect the cards played via OpenCV.
Current Project State
embedded_graphics driverReflashed and Updated Beaglebone Green
