@ShauryaMathur

# StarkMouse - High-Performance Hand Tracking System
AP Computer Science A Final Project

StarkMouse is an advanced gesture-controlled cursor controller that allows users to operate their computer mouse hands-free. Using MediaPipe and OpenCV on a native C++ backend combined with a Java Swing overlay HUD, the application tracks hand movements, applies coordinate smoothing, and detects finger-pinch gestures to trigger mouse actions.

---

## 1. How to Use the Program

### Prerequisites
- **Java Runtime Environment (JRE) / JDK 17 or higher**
- A working **Webcam**
- Windows OS (64-bit) for native dynamic library compatibility (`stark_mouse_jni.dll` and `opencv_world3416.dll` must be in the project root directory)

### How to Build and Run
To build the project and execute the `MainApp` entry point:
1. Open a terminal in the root directory.
2. Compile and package the application:
```powershell
mvn package
```
3. Run the generated executable JAR containing all dependencies:
```powershell
java -jar target/stark-mouse-1.0.0-jar-with-dependencies.jar
```

### Controls and Gestures
- **Mouse Movement**: Hold your hand in front of the webcam. Your cursor will follow the position of the **base of your middle finger** This anchor was specifically chosen to prevent cursor jumping when fingers pinch.
- **Left Click / Click-and-Drag**: Pinch your **Thumb and Index Finger** together. Releasing the pinch releases the mouse button.
- **Right Click**: Pinch your **Thumb and Middle Finger** together.
- **Mouse Scrolling (Scroll Mode)**: Pinch your **Thumb and Pinky Finger** together. While holding the pinch, move your hand vertically:
- Moving your hand **up** scrolls up.
- Moving your hand **down** scrolls down.
- While scrolling, mouse cursor movement is locked to prevent cursor drift.
- **Toggle HUD Overlay (Ctrl+Shift+G)**: Press this global hotkey to show or hide the Swing HUD window displaying the camera feed and landmarks.
- **Repositioning HUD**: Click and drag anywhere on the HUD panel to move the floating window across your desktop screen.

---

## 2. Problems Encountered During Development

- **The "java being anti cv" bottleneck**: Initially we planned on using a method called convex hulling to detect fingers as before libraries such as mediapipe that was a common finger detection method. We then realised that convexHull was terrible and we became primary sources as to why libraries such as MediaPipe have been developed. We then tried to bend the rules of the project by calling a python script running mediapipe detection. Our idea was quickly shut down. Our final option was taking the c++ library that was wrapped for python and creating a Java Native Interface from it using a tool called bezelisk. This process took a total of 5 hours, 10 gigabites of Shaurya Mathur's hotspot and our mental stanities. Once we finally got bezel instealled and working with our library we learned that c++ itself is annoying. Initially, running the Mediapipe tracker and the JNI JVM invocation on the same thread blocked the Swing event thread, resulting in a frozen UI. We resolved this by spawning a separate native execution thread (`std::thread`) in the DLL, calling `AttachCurrentThread` to register it with the Java VM, and passing frames asynchronously.