Devlog 2

Added Scrolling Functionality

Today I mapped the scroll wheel with pyautogui to scroll as I move my hand up or down while keeping my finger and thumb in contact.
To do so I calculated the mid point between my thumb and finger. This point acts as the main point whose position is considered for scrolling. You can see it in the video as well, it turns red when my fingers come in contact
Next to actually scroll I use the following code ->

diff = ti_mid[1]-prev_ti_midpt_y                    
if abs(diff) > small_movement:                        
    pyautogui.scroll(int(diff*sensitivity))
prev_ti_midpt_y = ti_mid[1]

Here I Compare the position of the mid pt in each iteration, calculate the difference and multiply with a constant which acts a sensitivity. Finally we use pyautogui to scroll it by that amount. Also if the magnitude of diff is too small (ie. less than small_movement variable) then the movement is not considered.

I also defined these variables at the top for easier configurations

sensitivity = 4
small_movement = 5
tap_threshold = 0.3
hand_detection_confidence=0.2
tracking_confidence=0.1
scroll_mode = True

Tap threshhold here is how close the fingers have to be to consider them in contact

Further Plans

Next I will be adding the mouse movement. I tried to implement that today but its gonna take more time so I’ll continue it in the next session. For that reason only I also added the variable, scroll_mode

Devlog 1

Intro

I have some plans for this project. I want to build a software which will let me control my pc from my hands via gestures.

Functionality

• Left click - Tapping Thumb and Index finger
• Right click - Holding thumb and index finger still while in contact
• Moving cursor - Moving the hand with the thumb and finger close by
• Scrolling - Holding the thumb & middle finger and moving it up and down
• Middle click - Tapping thumb and middle finger

Stack

• Python
• Mediapipe

Today’s work

Today I had setup a git repo and started writing the code. Till now I have made a basic structure which captures input via opencv, makes a dictionary with coordinates of all points in a hand and finally calculates the distance between thumb and index finger. Then it also detects if the thumb and middle finger are in contact.

However I faced one issue when calculating the distance. when I move forward and backward, the distance between 2 fingers is constant in the real world but it keeps changing for the camera as I move hand closer and further from the camera.

To fix this we understand that the ratio between the distance of my wrist and the point right where the pinky finger starts and the distance between fingers is always constant (because the palm never bends so it gives us a reference point to calculate distance), even when I move hand closer or far away. so that’s what I did, I calculated the ratio and used it as the distance which you can also see in the video below. There is also score which is the confidence of hand tracking and contact which tells if the index and thumb are in contact.