I am enhancing hand tracking now, and it’s starting to look good!

Here is how:

I use MediaPipe by Google to analyze image from front camera. MediaPipe Hand Landmarker returns hand landmarks in camera-relative format in two dimensions, so I need to make depth (z-coordinate).

I measure distance between wrist and fingers, in 2D space this means distance from camera (the farther away they are, the smaller they get).

Now, they are 3D, but still relative to camera. Then I simply send both hands as UDP packet for the driver.

The driver (PC side) computes real-world coordinates of the hands using saved earlier sent from ARCore and pushes them into Monado.