Dev log #5

Testing
I didn’t code much but I just ran my program across multiple days, and tweaked the prompt to make the categorizer more accurate. I think I’ve tweaked it to a point where it correctly categorizes daily activities done on a computer, and rarely gives a incorrect status.

Updates

• Added a examples folder to add training data along with a description for each image
• Added more specific constraints to the prompt like if 3 screenshot are exactly the same, user is idle
• Fixed the status if statement. Before it checked for the mention of the categories STEM, GAMES, STUDY, DOOM_SCROLLING, and MEETING, in that order. But now it checks the boxed phase for the category.

What’s next

• (UX) Packaging this program up for others to easily run
• (Data tracking) Track how much time a user spends in a certain category each day because people like to see data.

Dev log #4

I did some looking into prompt engineering, and I’m realizing it’s tough. Sometimes the AI, mistakes things I’m doing and puts them in the wrong category. Ex, if I had Discord open, and a friend sends me a video, the AI would mark that under doom scrolling, even though I didn’t do it. I’m also trying to keep it simple because sometimes when I tell it something specific, it thinks all cases should be considered the same way.

Updates

• Refined the prompt
• AI now looks through 3 of the most recent screenshots to determine what I’m doing.

Issues

• The AI still gets what I’m doing wrong sometimes.

What’s next

• Have it just run while I’m using my computer, and check my status every once in a while, and refine my prompt based on what the AI reasoned.

Dev log #3

So I got it working!!! I tried it out in Discord for a few hours, and the AI usage limits is holding up well. I was about to switch to a different AI because there were usage limits, and I kept hitting it. I realized it was because I didn’t have to upload the images to the Gemini cloud. I just let my program read them directly. And I made it so it takes a screenshot every 2 minutes instead of 20 seconds.

Updates

• AI now takes screenshots every 2 minutes
• AI Processes screenshot, and depending on the response and prompt, changes the status variable
• Connected AI to Discord program (Sends new status to bot after analysis)

What’s next

• The AI isn’t the best at differentiating what I’m doing right now, but I’m also not sure how good it is. I need to find a way to stress test it and teach it what screenshots are in what category. This also includes doing better prompt engineering.

Dev log #2

So I started testing out the AI here because I’ve never worked with it, and it wasn’t as hard as I thought it’d be. I decided to use a Gemini API, which I had to get from the Google AI Studio API Keys page. Then they also had a page somewhere teaching you how to use the API, which helped a lot (Gemini API help). The biggest issue I had was which model of Gemini I was using. 3.5 lite allowed more usage, and was the latest model that I could use.

Updates

• I made a test file for testing the pyautogui to take a screenshot and save it.
• Got a Gemini API key
• Learned how to make a hidden file called .env(environmental variables) and gitignore(Put the name of files you don’t want to commit) to hide my API key when I commit it to my GitHub
• Created a basic AI interaction with Gemini to take a text input, and an image input to generate a response.

What’s next

• Tune the prompt a bit to generate the response in a specific format(Right now I get an explanation sometimes, and not others)
• Connect this to the application display I made last time to show what I’m doing right now.

Check out the code below to see how my simple AI interaction works.

Dev log 1

This is the start of a mini program I’m making that’s going to track my activities on my computer to give me an incentive to be productive! So below here I have a plan of what I’m trying to build, and it seems I’m going to have to learn about using APIs, communication between different local programs like IPC.

I’m not sure how to track the time I spent planning so I started a bit of code to log this.

Development

• I had to make a app in discord to act as my status display, and this gave me a app ID
• Then I got a test script and tested to see if I could connect my script to the app using app ID. I had to install the pypresence plugin, which is how, python can communicate with discord.

What’s next

I have to figure out what AI I could use, and learn how to incorporate a AI in a python script.

I’m calling it Nozi bot for now because it sounds like “nosy” and it’s like tracking what you’re doing for your nosy friends.