Work on hyperparameter tuning

Trained for a day, bad results due to poorly optimized hyperparameters

Added rewards for winning rounds

Street Fighter II Model - Model Regressing

After training for another night, my model starting regressing again, now even losing to Guile

Next steps

The issue is once again my reward function, since its rewarding the model based on its damage, it would have a higher reward if it loses one round out of the three, since it would be able to deal more damage to bloat the rewards. My plan is to tweak the reward function again to incorporate rewarding it when it wins any rounds.

Street Fighter II Model - Beaten Ken!! (Seconds level)

After training for a bit more than a day, I was finally able to beat Ken, and move on to the next level to Zangief! (The car is sort of an intermission level)

Street Fighter II Model - Beaten Ken!! (Seconds level)

After training for a bit more than a day, I was finally able to beat Ken, and move on to the next level to Zangief! (The car is sort of an intermission level)

Street Fighter II Model - Finally beats Guile!! (The first level)

After around 10 hours of overnight training (17M timesteps/frames), the agent can finally beat Guile and reach Ken, but it still falls a bit short beating Ken.

Reward bug

This was the major bug that was making the model not even damage Guile before.

Health resets from 0->176 when the model loses on the first round.

This makes the reward become:

old frame health - new frame health = damage taken
0 (dead) - 176 (max health) = -176 
reward = damage_dealt - (damage_taken)
reward = damage_dealt - (-176) = damage_dealt + 176

Dying seemed like a large reward to the model, so it learned that dying was good instead of damaging Guile.

The Fix

damage_dealt = max(0, old_enemy_hp - new_enemy_hp)
damage_taken = max(0, old_player_hp - new_player_hp)
reward = damage_dealt - damage_taken

This made it so that if any one of the players died the frame where it recovers health, it would return 0. Since the max function would return the largest value from the parameters given.

For example:

print(max(0, -176))

This would print:

0

Street Fighter II Model - Finally beats Guile!! (The first level)

After around 10 hours of overnight training (17M timesteps/frames), the agent can finally beat Guile and reach Ken, but it still falls a bit short beating Ken.

Reward bug

This was the major bug that was making the model not even damage Guile before.

Health resets from 0->176 when the model loses on the first round.

This makes the reward become:

old frame health - new frame health = damage taken
0 (dead) - 176 (max health) = -176 
reward = damage_dealt - (damage_taken)
reward = damage_dealt - (-176) = damage_dealt + 176

Dying seemed like a large reward to the model, so it learned that dying was good instead of damaging Guile.

The Fix

damage_dealt = max(0, old_enemy_hp - new_enemy_hp)
damage_taken = max(0, old_player_hp - new_player_hp)
reward = damage_dealt - damage_taken

This made it so that if any one of the players died the frame where it recovers health, it would return 0. Since the max function would return the largest value from the parameters given.

For example:

print(max(0, -176))

This would print:

0

Coding Hours: 2h

Huge reward function bug fix

After training my model for 5 million timesteps (basically 5 million frames), I realised a huge flaw in my reward function logic.

It would award the players a huge reward after dying once and their health bar replenishing, since I failed to catch the specific case.

This was why the previous models weren’t performing well (see the video below)

I’m going to train my new model with the reward function fixed overnight today and see how it goes!

Coding Hours: 2h

Huge reward function bug fix

After training my model for 5 million timesteps (basically 5 million frames), I realised a huge flaw in my reward function logic.

It would award the players a huge reward after dying once and their health bar replenishing, since I failed to catch the specific case.

This was why the previous models weren’t performing well (see the video below)

I’m going to train my new model with the reward function fixed overnight today and see how it goes!

Finished hyperparameter tuning

After running the tuning program overnight, I’m going to use those parameters to actually train the model overnight to see how it performs

Finished hyperparameter tuning

After running the tuning program overnight, I’m going to use those parameters to actually train the model overnight to see how it performs

Coding Hours: 2h

Optimised model even further

• Stripped down the game window to a way smaller 84x84 resolution and also grayscaled the game. Also set up 4 parallel environments for the model to be able to train even faster than before.

• Started hyperparameter tuning before training the actual model to ensure it is able to train at the most effective and efficient way.

Below is what the AI uses to train (example not the actual windows, they would be scaled down and be grayscaled so it would be easier to process for the model):

Coding Hours: 2h

Optimised model even further

• Stripped down the game window to a way smaller 84x84 resolution and also grayscaled the game. Also set up 4 parallel environments for the model to be able to train even faster than before.

• Started hyperparameter tuning before training the actual model to ensure it is able to train at the most effective and efficient way.

Below is what the AI uses to train (example not the actual windows, they would be scaled down and be grayscaled so it would be easier to process for the model):

After Training for another million steps overnight, my model has now finally learnt to crouch and block, while it still doesn’t deal any damage, it is still an improvement. This model right now is very unoptimised, I’m working on an improvement to optimise everything to make training a lot faster and way more efficient.

After Training for another million steps overnight, my model has now finally learnt to crouch and block, while it still doesn’t deal any damage, it is still an improvement. This model right now is very unoptimised, I’m working on an improvement to optimise everything to make training a lot faster and way more efficient.

Coding Hours: 1 Hour 50 Mins

Started making a custom reward function and also optimised the learning environemnt so that the model can learn more efficiently

Coding Hours: 1 Hour 50 Mins

Started making a custom reward function and also optimised the learning environemnt so that the model can learn more efficiently

Trained model overnight again, but it still showed little improvement. I will try to create an environment and setup the reward system so it can train more efficiently and also host train it on some free cloud websites rather than my own laptop.

Trained model overnight again, but it still showed little improvement. I will try to create an environment and setup the reward system so it can train more efficiently and also host train it on some free cloud websites rather than my own laptop.