Cue
By
@L0stsky
- 10 Devlogs
- 22 Total hours
Tool to Make Making Guides Easier!
So I noticed that I had put the AI usage to None in the beginning but later i used it to find some bugs etc and make the startup scripts so heres my AI usage if you wanna know
So I Broke it Again , the Models were Broken So i Just Updated the Models and Increased The Number of API Tries to 15
So I Found out that One of the Model I was Using Was Not Capable of the Task 🥀🥀, So I Updated the code to Use New Capable Models and also Retry If the Model Could not Extract the Data. And Now I gotta Wait Till Tommorrow Till I can ReMake the Windows Exe File (I use Linux soooo) 😭😭😭
So I was Finally Able to get Pyrunner Working and Make the Godamm executables for Linux and Windows.
Well Finally after 22 Hours of Working and a dead Pen Drive (Idk why tf its Firmware Got erased) I finally Completed Cue
Ship #1
Pending review
So I Made Cue - A Multi OS Compatible Tool which Allows the user To Make Guides which is as Simple as Runing a Script in the background, Doing Whatever You want to Record, Use CTRL+CLICK or CTRL+ENTER to record the steps and Done!!
So Basically Heres how it Works ->
The User runs a Startup Script
The Script Takes a Screenshot Everytime the User Presses CTRL+CLICK or CTRL+ENTER and Saves it.
After going through all the Steps, the User Presses CTRL+ESC to End the Recording.
After this, a New Script Is launched which loads in these Screenshots and then Uses a VL (Vision Languag) Model to Extract the User Intent.
After going through all the imgs, the Data Extracted is passed through a more “Intelligent” Model which Compiles it Into a Guide.
The QOF Improvement is that the It Removes the Boring task of telling someone “press x and then Click Y” till they finally Get it
The Major QOF improvements are :
Zero manual Screenshoting or Cropping
No Manual Writing Steps
Easy to Follow Guides and Allow Screenshots to be Directly Attached.
No Manual Making of python Environment etc.
So the Challenging Part was to get the Fricking Wayland Screenshot Working Cuz wayland wont allow Some script to Take Screenshots. Also another thing was Getting the Models to Cooperate and Stop “Hallucinating” Random Output
The thing I’m Proud of the Most is that I was Able to Get the Wayland Screenshot Working and Also How I implemented the Paralled AI model Contacting Architecture for speed.
So To try Cue out, You Just Click Try Project it will download a ZIP file, Extract that and Open the Folder. In the Folder Run “Cue_Windows.bat” for Windows and “Cue_Linux-Mac.sh” for Linux or Mac.
Ensure that python is Installed and For Mac The Program has Permissions (In more detail on github Readme)
DevLog_6
So My dum Brain found out that Python Envs cannot just be “Packaged “ and Used on another Computer Sooooo I Updated the Startup Scripts to Make One On startup.
Also IDK why but my dummahh tried to Retrain a local 2B pretrained Model but after getting 1 Epoch in 2 HOURS I just Lost Hope and Dumped the Idea.
Devlog 5:
SO If you don’t Remember, Im Working on CUE -> A Easy af To use Tool to Make Easy to Follow Guides which Is simple as clicking to Run a Script and Then Doing Whatever Task You want. The Script AutoMatically saves Your actions and After You are done, these are Compiled into a Text Based or Both text and Image Based Guide of Your Chosing!!!
Here’s What Im Working On:
Made a Single Click Startup Script.
Added a API key Verification.
So I Fixed Some Major Cross Platform Code Errors.
Packaged a Python Env With the Things for Ease of Use.
Tried and Tested On Linux and Windows Thought I couldnt On Mac Since I dont Use that.
Now I plan to Use the Tool to Make Guides Regarding How to use it!!!!
DevLog_4:
Finally Got the Fricking Images in the Guide to Work and The Sudo Permission to Work Correctly.
First I tried to Embed them Directly But that didnt Work so i Just Put the Images Along with the Guide and Used Paths to render them.
Then I made Improvements to my Screenshot and the Backend Code Like Gracefully exiting Rathar then the Forced Exit I was Doing Earlier.
Added a “NOTE to the AI” Feature and Introduced a Smarter Model for Compiling the Data into a Guide.
Now I plan to Use Pyinstaller to Make it a Executable and Make the Model Format the Guide a Little More Beautifully.
DevLog_3
I have Attached a Demo Video of the Current Project.
In the Video You Can see that the Output File is Not Properly Formatted because I opened it in Notepad But in The Actual Docs Viewer it Looks Fine. Also the |Image| Tags is a Beta Feature and Is not Fully Implemented.
Also If you are Wondering, it Took the Script like 2 Minutes to Process the Screenshots (I have Cropped the Video).
So I Finally Completed The Scripts for Running the Project.
I made 2 scripts -> For the GUI version and the CLI version.
I was able to make a minimal GUI Setup for the Project though its still a little buggy which I’ll Fix Later.
Now I am Working on Fixing Some Issues with the API, Fully Implementing the images in the Guide Feature and Trying to turn it into a Application Rather than running it via Scripts.
DevLog_2
So I Finally Was Able to Complete the AI Backend Part of the Project and let me tell you - IT WAS A NIGHTMARE.
For the First 4 Hours I Tried to get a Local VLM to Work but Due to my Low end Pc it wouldnt Work so I just Gave up and Switched to Free Openrouter Models.
So Here is What I’ve Made :
I Made the Python File Which would Load in all the Screenshots from the Main Script.
These Screenshots are then sent to an API which Describes What the User is Doing by Using the Cursor Locations.
All these Observations are then Compiled by a Smarter Model To make a Comprehensive Guide.
Now I plan to Try to Make a Simplistic IN-Python UI and a Script to Do all the Tasks and Package it.
I’m building CUE—an AI-powered guide generator to solve the headache of manually making step-by-step instructions. The goal is to have a seamless tool that just works across all OSs.
What’s done so far
Finished the main code and the specific OS capture handlers.
Every time you click anywhere on the screen, it automatically grabs a screenshot and dumps it into a local folder.
The Wayland Pain: Wayland’s OS security made capturing clicks a nightmare compared to every other platform. I basically had to put Gemini’s black cap on to help me hack around Wayland security and get those screen captures working.