@utilisateur_richard
on
Pokémon MuZero
This is an AI (neural network, please do not think it is like ChatGPT) designed to reach superhuman / master level in PTCG (the Pokémon Trading Card Game).
Unfortunately, due to its nature and due to the fact that the game engine is proprietary, I cannot send it to you and let you play with it.
It is based on the MuZero architecture (made by DeepMind), on the Stochastic MuZero, on the Gumbel MuZero architecture, and on the EfficientZero architecture.
How does it work? It is 3 NNs (neural networks) that work together to be invincible at a given game without even knowing the rules!
How can it play without knowing the rules?
Well, it has an imagination that lets it predict what will happen all by itself without any help.
For that, the first network, the Encoder formerly named h, will create a latent space (the imagination / position) based on the real state of the game for the other networks.
After that, the Strategist (formerly f) network will see the latent space (the imagination / position) and check:
- Is this position a winning or a losing position with the V (value) network?
- What will be the next best moves with the P (policy) network?
Then, based on the moves suggested by the P network, the the Simulator network (g) simulates the outcome to create the next imaginary state (latent space). It repeats this loop with the Strategist (f) and the Simulator (g) until it exhausts its thinking budget.
But does it just imagine ONE line of play and blindly follow it? No! It actually imagines many different possible moves at the same time, like a tree of possibilities (this is called Monte Carlo Tree Search, or MCTS). At each node of this tree, the model has to pick which branch to expand next, and it does this using a formula (called PUCT) that balances two things: how good the P network and V network think this branch is (exploitation), and how few times this branch has been visited so far compared to its siblings (exploration). So a branch gets visited more when it keeps looking promising as it gets expanded deeper, but every branch still gets a fair shot early on so the model does not tunnel-vision on its first guess. This creates a feedback loop: good branches get explored more, which refines their value estimate even further, and bad branches get abandoned early. In the end, the move it picks is the one that got explored/visited the most during this search, not just “whatever the P network liked the most” at first glance. This is what makes the whole system so much stronger than the raw network alone!
Safety mechanics:
To avoid having problems like the imagination of the model being completely inaccurate, I added some probes (they are networks, but just remember that they are here to evaluate the model). Their goal is to extract information from the latent space to see if it is accurate. E.g., we create a latent space in which a Pokémon is or isn’t (50-50) K.O., and the probe has to, based only on the latent space, say if the Pokémon is K.O. or not.
I also added a lot of logs, for example a log that counts the number of actions that the model takes (e.g., does it attack or does it pass its turn? (I had a problem one time where the model passed its turn 75% to 90% of the time!!)).
below, you can see two different latent space (imagination), every dot is a real position in the game, the distance between the dots is their similarity (far = different, close = similar), you can also see in the 4th graph the age of the state (blue = start of the game, yellow = end of the game) and the most important in the 1st, 2nd and 3rd graph, you can see the predicted value of the model, the real value (red = win blue = loose) and the difference between the prediction of the AI and the real value
the first png is the model at the first state (not trained) and the second png is the model at its best level
