Rocket Landing DQN

After an extensive debugging session, the project is now complete (this time for real).

The main problem was the model was exploring too much. Even though it knew what to do, it was just refusing to use its knowledge so I made the epsilon decay faster so it switches to exploitation earlier.

After this I had to find the sweet spot number of episodes to ensure consistency in its trials. This was found to be 3300 after observations based on trial and error.

Unfortunately, since DQN is not very stable, there will be a difference in quality whenever you run the program. Some landings may be smooth, others may be…questionable.

I had a lot of fun with this project and I encourage anyone that wants to get into reinforcement learning to check it out.

There is no “interactive demo” since its a simulation. So check out the code on github and I will attack this video to the project page as well.

I realised I had some logical errors in my step function which I have now fixed:

Previously, the function checked whether the rocket landed, crashed or was in the air then it implemented changes to the rocket’s y coordinates via the velocity depending on the action taken

This order of working is wrong and should have been reversed. The check for landing, crashing or flying should be implemented after the rocket’s y coordinates have been updated in accordance to the action taken.

The AI has now overcome the problem, hence, I must increase the complexity.

I have done this by introducing acceleration due to gravity and a fuel system that prevents the rocket from acceleration if it runs out of fuel. This forces the AI to think carefully about using its thrust rather than exploiting the previous infinite fuel.

With this implementation, the rocket landing is much more realistic and smoother.

The DQN model can now successfully land the rocket after the latest debugging session.

Fixed:
IF statement conditions for rocket’s y coordinates and ground as I forgot that 0 is the sky so the rocket’s y coordinate must be greater than the ground and its velocity must be a safe speed for it to land successfully
Terminated or truncated flag in step function for landing successfully was originally false which is incorrect as the algorithm must end since the goal has been achieved
Order of IF statements for crashing and landing checking in the GUI loop was wrong as even if the rocket landed, it would continue to fulfill the conditions for a crash and so the check for the landing conditions must be placed before.

Next steps:

I am curious to see how the agent will manage landing the rocket if I introduce time dependent velocity as a result of acceleration due to gravity that changes as the simulation progresses.

Its clear the DQN model is having some issues.

The reward fluctuates and never reaches positive which means the rocket never lands in any episode.

To fix this I lowered the learning rate and decreased the penalty for crashing.

I also fixed the GUI logic so that if the velocity is a low speed and the rocket’s y coordinate is the same as the ground then the rocket does not crash and lands.

However, this issue persists and so I will now examine both the GUI and the AI logic to find the source of the error.

I have completed the rest of the code which involved coding the training process and initialising the policy and target networks.

A challenge I am facing now is modifying the code to suit my specific environment and updating the GUI to visually showcase the result of the AI’s choices.

A few changes I made were:
Added gravity and rocket acceleration variables
Removed the previous actions and introduced only two choices for the model which is either to apply thrust or choose to not apply thrust.
Fixed the returns from the step function so that it returns a numpy array that contains the rocket’s y coordinate and the rocket’s velocity
Regenerated the AI generated code to include a crash animation

The specific challenge with the modifications is that the input requirements are very specific and the manipulation of these inputs relies on complex functions.

What are the next steps:
I will use AI to debug my codes, specifically the errors associated with incorrect input and output dimensions and structures. I will also use it to debug my GUI screen.
Once the model is working, I will change the acceleration to make it relative to time and introduce a time variable to reflect acceleration due to gravity.

I realised I made a mistake in my previous implementation of the DQN model.

I forgot to inherit from the required pytorch module and I forgot to feed the state_size and action_size to the constructor method.

To further evaluate and fix my code, I will be using an implementation from GeeksForGeeks to guide me while modifying some sections to suit my specific scenario.

I have implemented the neural network that will be used by the Deep Q Network model.

As the task is not very complex, I decided to utilise 32 neurons. In the constructor method, I am essentially stating that one piece of data will be given to the 32 neurons (y coordinate) and I expect two q values to be outputted (as the rocket can only go up or down).

Used Gemini to generate the GUI so that I can focus on the actual Deep Q Network

After that I implemented a velocity variable that manipulates the rocket’s y coordinates

Finally, I implemented a reset function and a step function which changes the velocity of the rocket based on the AI’s action and it also returns required data for training (Y coordinate, reward, Done, Truncated, Extra Info)

To reduce the complexity of the problem, I am ignoring horizontal movement (x direction) and focusing only on the vertical movement while ignoring factors like air resistance.