Street Fighter II Model - Finally beats Guile!! (The first level)

After around 10 hours of overnight training (17M timesteps/frames), the agent can finally beat Guile and reach Ken, but it still falls a bit short beating Ken.

Reward bug

This was the major bug that was making the model not even damage Guile before.

Health resets from 0->176 when the model loses on the first round.

This makes the reward become:

old frame health - new frame health = damage taken
0 (dead) - 176 (max health) = -176 
reward = damage_dealt - (damage_taken)
reward = damage_dealt - (-176) = damage_dealt + 176

Dying seemed like a large reward to the model, so it learned that dying was good instead of damaging Guile.

The Fix

damage_dealt = max(0, old_enemy_hp - new_enemy_hp)
damage_taken = max(0, old_player_hp - new_player_hp)
reward = damage_dealt - damage_taken

This made it so that if any one of the players died the frame where it recovers health, it would return 0. Since the max function would return the largest value from the parameters given.

For example:

print(max(0, -176))

This would print: