@Rayed_Ahmed

A lot of changes have been made since the last log:

I downloaded an open source LLM called Olama that I imported and used to generate flashcards from notes

I utilised a library called python-to-docx to convert google documents into text files which can be fed into the AI.

I created a form that allows users to attach files and name their flashcards which will then be passed into the python-to-docx code and finally, the AI.

Currently, I am improving my prompt and figuring out how I can efficiently and effectively split the AI’s response (flashcards) into terms and definitions.

I told the AI to make flashcards with new lines betwene new cards and colons between terms and definitions and I have been trying to utilise python’s split function to sort the data however, the prompts are not satisfactory.

I will look towards getting gemini to engineer a prompt for me and maybe finding a more efficient way to sort the data.

So far I have been working on the basic functions before actually working on the AI flashcard generator function.

I started by creating routes and functions for my different webpages (home, library, view). After this, I created a base html file that I could extend into the rest of my html files which essentially lets me have a uniform interface to work on.

Finally, I created my template files which are basically dynamic html files that display different data based on the id being passed into the URL. In this case, I have my data in a file called data.py where id represents the key in my dictionaries. If you click on a flashcard set, its id is passed into the URL and the template file renders based on that information.

I also did a bunch of css stuff where I used AI to help me figure out how I could create the flipping animation for my cards. Other than that, I created blueprints for buttons.

My plan for my next coding session is to start working on the AI flashcard generator function which will involve importing libraries and creating a framework to read google doc files and store them in a string. In addition to this, I probably will have to figure out how to use API keys and prompt engineering to get the best results.

After an extensive debugging session, the project is now complete (this time for real).

The main problem was the model was exploring too much. Even though it knew what to do, it was just refusing to use its knowledge so I made the epsilon decay faster so it switches to exploitation earlier.

After this I had to find the sweet spot number of episodes to ensure consistency in its trials. This was found to be 3300 after observations based on trial and error.

Unfortunately, since DQN is not very stable, there will be a difference in quality whenever you run the program. Some landings may be smooth, others may be…questionable.

I had a lot of fun with this project and I encourage anyone that wants to get into reinforcement learning to check it out.

There is no “interactive demo” since its a simulation. So check out the code on github and I will attack this video to the project page as well.

I realised I had some logical errors in my step function which I have now fixed:

Previously, the function checked whether the rocket landed, crashed or was in the air then it implemented changes to the rocket’s y coordinates via the velocity depending on the action taken

This order of working is wrong and should have been reversed. The check for landing, crashing or flying should be implemented after the rocket’s y coordinates have been updated in accordance to the action taken.

The AI has now overcome the problem, hence, I must increase the complexity.

I have done this by introducing acceleration due to gravity and a fuel system that prevents the rocket from acceleration if it runs out of fuel. This forces the AI to think carefully about using its thrust rather than exploiting the previous infinite fuel.

With this implementation, the rocket landing is much more realistic and smoother.

The DQN model can now successfully land the rocket after the latest debugging session.

Fixed:
IF statement conditions for rocket’s y coordinates and ground as I forgot that 0 is the sky so the rocket’s y coordinate must be greater than the ground and its velocity must be a safe speed for it to land successfully
Terminated or truncated flag in step function for landing successfully was originally false which is incorrect as the algorithm must end since the goal has been achieved
Order of IF statements for crashing and landing checking in the GUI loop was wrong as even if the rocket landed, it would continue to fulfill the conditions for a crash and so the check for the landing conditions must be placed before.

Next steps:

I am curious to see how the agent will manage landing the rocket if I introduce time dependent velocity as a result of acceleration due to gravity that changes as the simulation progresses.

Its clear the DQN model is having some issues.

The reward fluctuates and never reaches positive which means the rocket never lands in any episode.

To fix this I lowered the learning rate and decreased the penalty for crashing.

I also fixed the GUI logic so that if the velocity is a low speed and the rocket’s y coordinate is the same as the ground then the rocket does not crash and lands.

However, this issue persists and so I will now examine both the GUI and the AI logic to find the source of the error.

I have completed the rest of the code which involved coding the training process and initialising the policy and target networks.

A challenge I am facing now is modifying the code to suit my specific environment and updating the GUI to visually showcase the result of the AI’s choices.

A few changes I made were:
Added gravity and rocket acceleration variables
Removed the previous actions and introduced only two choices for the model which is either to apply thrust or choose to not apply thrust.
Fixed the returns from the step function so that it returns a numpy array that contains the rocket’s y coordinate and the rocket’s velocity
Regenerated the AI generated code to include a crash animation

The specific challenge with the modifications is that the input requirements are very specific and the manipulation of these inputs relies on complex functions.

What are the next steps:
I will use AI to debug my codes, specifically the errors associated with incorrect input and output dimensions and structures. I will also use it to debug my GUI screen.
Once the model is working, I will change the acceleration to make it relative to time and introduce a time variable to reflect acceleration due to gravity.

I realised I made a mistake in my previous implementation of the DQN model.

I forgot to inherit from the required pytorch module and I forgot to feed the state_size and action_size to the constructor method.

To further evaluate and fix my code, I will be using an implementation from GeeksForGeeks to guide me while modifying some sections to suit my specific scenario.

I have implemented the neural network that will be used by the Deep Q Network model.

As the task is not very complex, I decided to utilise 32 neurons. In the constructor method, I am essentially stating that one piece of data will be given to the 32 neurons (y coordinate) and I expect two q values to be outputted (as the rocket can only go up or down).

Used Gemini to generate the GUI so that I can focus on the actual Deep Q Network

After that I implemented a velocity variable that manipulates the rocket’s y coordinates

Finally, I implemented a reset function and a step function which changes the velocity of the rocket based on the AI’s action and it also returns required data for training (Y coordinate, reward, Done, Truncated, Extra Info)

To reduce the complexity of the problem, I am ignoring horizontal movement (x direction) and focusing only on the vertical movement while ignoring factors like air resistance.