So it’s been a while. In short: making the program recognize the difference between several single-finger motions and one multi-finger motion turned out to be quite difficult. My code originally discarded a finger’s action history once it triggered an action, which worked when the fingers were being tested separately, but when tested together ended up making the program count the fingers one-by-one, triggering their actions, before discarding the history so they couldn’t be analyzed for whether or not all the gestures were actually together. I solved this by adding another gesture queue that opens the first time something is detected, stays open for a processing window, then closes after no further inputs are received. I also implemented a hand preference feature so the program can only use one hand rather than looking for both. It does work, however it still trips up when the wrong hand enters frame first, and the right hand right after it. I also implemented a proximity checker to try and make the code only work when a finger is pointed towards the camera, but it worked poorly and I will likely not use it for the ship. I also tried to make a new processing mode where instead of the motions of all fingers being recorded individually and then aggregated and turned into a direction later, only the index finger motion would be checked initially, then fingers were categorized based on whether they moved relative to the index finger or not. Those that didn’t were assumed to be part of the gesture, and therefore counted. I thought this would make the program more reliable since it would only require the index finger to be tracked well and the settings could be fine tuned to perform best for just that finger, but it ended up being kinda annoying. The processing window had to be really large in order for it to actually capture relative motion, the relative positions were tricky since the program couldn’t differentiate well whether one finger was covering another, and overall it just performed a lot worse. I might try and edit it in the future, but for now the regular mode works just fine as long as there’s a good amount of light. So far 1, 2, and 3 finger gestures work as separate triggers, and I will likely work on clicking in some way. On my setup, 3 fingers changes the window, 2 fingers scroll, and 1 finger moves the mouse. It would be nice to make it continuous instead of moving a set amount every trigger, but it does work.
No complete update yet (working on giving the program a grace period when the hand landmarker model fails to process a frame or two), but I just wanted to share this gem of a debug log. I think I messed something up somewhere because…
ydotool integration now works! I’ll admit it’s a little choppy at the moment and there’s a lot of optimizations to be made, but it isn’t throwing any explicit errors which is an improvement. One issue I will have to deal with is it being incredibly slow, especially considering that it has a very similar function definition as the terminal command option. I think this is because it is compiling the config.json string into numerical identifiers every single time it triggers rather than doing it once in the setup stage as overhead and just referencing the numerical values as variables. I didn’t foresee this being an issue since there’s a maximum of 4 triggers (and therefore 4 compilations) per second, but I will continue to work on that.
Another thing that I definitely HAVE to work on is hand choosing. I originally tried to record my computer through my phone instead of screen recording so my actual hand would be captured (ergo, there would be proof I was actually controlling this with my hand) but the program kept locking onto the hand holding my phone and ignoring my other hand. Therefore, I’ll probably also be working on adding a config option to force the program to only recognize either the left or right hand.
Finished an installation script that downloads the necessary packages via pip if the user doesn’t already have them. It shouldn’t be too necessary because the packages will already be bundled in since I plan to use pyinstaller when I ship it, but it’s nice to have if some users prefer to run the code themselves or make their own modifications. There is a bit of a bug though: since this program needs both opencv-python and mediapipe, which itself needs opencv-contrib-python, there are two packages that share the ‘cv2’ namespace. This isn’t an issue if you have both, but if the user only has one of the two packages the program will fail because it expected both to be functional, but won’t catch why it failed because all it sees is that it was able to import cv2. Not sure how I’m going to fix that, but for 90% of people who use this it shouldn’t be of much concern.
Also the reason you can’t see my camera this time is because I’ve turned off the show feed flag, it kept getting in my way for testing. However, you can see via the “camera is active” notification that it did turn on.
Goodness gracious it is 3am…
I’ve made some more good progress on the script, including (amongst tidying up as well, of course), implementing a function that creates a new config file if it can’t locate one, a function that automatically downloads the correct hand landmarker model so it doesn’t need to be bundled into the directory by default, a better json loader function, support for absolute paths (which will become incredibly important when I ship this because relative paths can be hit or miss), an improved automatic selection tool to choose what framework should be used, and more.
I also changed how constants work: instead of being loaded one-by-one right before they’re needed within their functions, they’re all loaded in the setup phase as soon as the config file is resolved, and declared as global variables. This helped reduce a lot of redundancy in assigning variables.
In terms of bugfixes, I fixed a loading order error in which the program attempted to access config values before they were assigned to constants, a couple import errors (these were my fault somehow I managed to put import statements within an if statement), added a new error exception for when the config file is malformed, fixed a small read-write permissions error, and solved an initial issue I had with ydotool (whenever I would send a keystroke it would close the “shell” it was running in before the unpress command went through, and the keys would be permanently stuck with no way to reopen the shell to undo it. The only shortcut this affects now is closing a window intentionally, though I don’t think that’ll be a problem when it’s running minimized.)
So, funny story, I did not in fact fix it tomorrow,” I decided to take a break for the day and when I came back I made a series of poor system management decisions that ended in me wrecking systemctl and having to restore from a snapshot.
HOWEVER, I have gotten quite a bit done. Non-exhaustively: I completed the 5-finger tracking system so the program keeps track of all 5 fingers, including the wrist; I made it so that the program recognized ANY finger to trigger an execution rather than just the index finger; I implemented a debug flag in the config.json to show specific debug values (I will work on making it pretty later); I implemented a show view flag so the user can decide whether they want a camera frame active or not; and (perhaps the largest change) I am working on setup tool that can configure the program on the first run.
The reason I’m working on the setup tool now instead of saving it until the end is that, due to my last devlog where I decided to pivot to using keyboard shortcuts instead of just terminal commands, I’ve had to look for packages that can help trigger certain keystrokes. I’ve landed on pyautogui for the main package – it’s easy to use, supports windows, mac, and linux on x11, and isn’t too large. However (and this really annoyed me), Linux users on Wayland (aka me) need a completely different tool (ydotool) since wayland blocks pyautogui and pretty much every other python script from executing.
Therefore, instead of building the entire thing around ydotool and having to retrofit pyautogui support for the majority of people to be able to use my script, I’ve decided to develop both at the same time using a VM when I need to. It’s definitely taking longer than it would otherwise, but I hope the result will be much better.
Alright, so this devlog (of a whopping 20 minutes) isn’t really to highlight progress, but to highlight a decision I’ve made: up until now, I’ve having my script trigger a terminal command in order to execute actions. However, this is very setup-dependent, not cross platform, and a little overkill if we’re being honest. Instead of that, I’ve decided that it would be best to pivot to having movements trigger keyboard shortcuts, which means the program should (in theory) work across different operating systems without user hassle. I plan to use pyautogui for this, though I may also use pynput if it ends up being more efficient. Secondly, I have also decided to add an option to force the program to require that a finger be extended towards the camera in order to consider its motion, which may make multi-finger combinations easier to execute with less accidental triggers.
Good news and bad news: The good news is that I’ve been able to drastically clean up my code, make a few performance optimizations, and make the camera viewport respect config.json preferences. I’ve also been able to port 90% of my code over to using dictionaries and keys instead of just variables, which now allows it to store data about the position and movement of every finger instead of just one(although the index is still the only one being used right now).
The bad news is that I am truly losing my marbles trying to devise a system that would be able to wait until all fingers have been processed to evaluate whether or not a gesture was a one, two, or three finger one while still being in the normal execution loop, and not interfering with the next frame’s processing. Hopefully I’ll be able to come up with a solution tomorrow.
YESSSSSSS it works!!! It is incredibly janky, hypersensitive, and only
responds to the index finger right now (not to mention that it doesn’t
respect the config.json, that’s a later me problem), but hey, it exists!
Next steps are to clean up the code and add tracking for every finger
and the wrist.
Just posting a devlog for an initial update. Unfortunately I borked my hackatime config so it didn’t track the work I did initially but it’s fine because right now all I have is the json loader and most of the camera feel handling. The json took me an embarrasing amout of time because I’ve never touched json in Python code before.
