Devlog 1

Intro

I have some plans for this project. I want to build a software which will let me control my pc from my hands via gestures.

Functionality

• Left click - Tapping Thumb and Index finger
• Right click - Holding thumb and index finger still while in contact
• Moving cursor - Moving the hand with the thumb and finger close by
• Scrolling - Holding the thumb & middle finger and moving it up and down
• Middle click - Tapping thumb and middle finger

Stack

• Python
• Mediapipe

Today’s work

Today I had setup a git repo and started writing the code. Till now I have made a basic structure which captures input via opencv, makes a dictionary with coordinates of all points in a hand and finally calculates the distance between thumb and index finger. Then it also detects if the thumb and middle finger are in contact.

However I faced one issue when calculating the distance. when I move forward and backward, the distance between 2 fingers is constant in the real world but it keeps changing for the camera as I move hand closer and further from the camera.

To fix this we understand that the ratio between the distance of my wrist and the point right where the pinky finger starts and the distance between fingers is always constant (because the palm never bends so it gives us a reference point to calculate distance), even when I move hand closer or far away. so that’s what I did, I calculated the ratio and used it as the distance which you can also see in the video below. There is also score which is the confidence of hand tracking and contact which tells if the index and thumb are in contact.