@ttomf

Hi! So I’m here again after many exams (I didn’t have time and strength to iteratively test this ) and I’m gonna make this better using better Hand Landmarkers and (hopefully) run on Window$.

I think it’s time to new release! I merged upstream, created tag, built app and driver and published on GitHub.

I am working on implementing hand controller using same data as hand tracking does. It also includes the “aim pose” - where does controller point. On video, it’s shown as red line, I take it from the index finger. I hope I will be able to play some better VR games!

I’ve fixed some bugs and leaks, added more configuration to Android app and created hand tracking vizualization for the PC script. Also, I am working on integrating hands as controllers (because controllers have better support across games).

YEAH! I have finally done the PC camera hand tracking! It has much better FOV than phone camera and more computing power.

It’s just Python script with MediaPipe, sending the data to 127.0.0.1 to hand port, and it has same format, so the driver doesn’t even know the data is sending someone other than phone. You will ofc need to disable phone hand tracking, otherwise they will collide.

As you see in the video, I can move the hands far and they still get tracked. (End of the video is when I took off the headset - it was also tracked :D)

Now I am making just minor changes to the driver as the base is nearly done. I added screen size synchronization with phone, so the driver now can send right stream. Also I am making small bugfixes and improvements.

I added settings on phone, so now you can really configure the ports on both sides!

Also there’s option to disable hand tracking or 6DOF (then it will send only rotation of headset)

I created simple terminal installer in python that downloads and setups Monado Phone Driver.

Thanks also for the positive reviews and the 19.12x stardust multiplier — when I started this project, I didn’t think I’d ever finish it :D

Last devlog before ship! I just merged upstream monado, built release binaries and created GitHub release.

So I finished the hand tracking! I tried to make it as good as possible on a phone, and it looks like it might actually work! Even though I’m limited by the hardware, you can see in the video that it tracks pretty well.

(video from openxr-simple-playground)

I am enhancing hand tracking now, and it’s starting to look good!

Here is how:

I use MediaPipe by Google to analyze image from front camera. MediaPipe Hand Landmarker returns hand landmarks in camera-relative format in two dimensions, so I need to make depth (z-coordinate).

I measure distance between wrist and fingers, in 2D space this means distance from camera (the farther away they are, the smaller they get).

Now, they are 3D, but still relative to camera. Then I simply send both hands as UDP packet for the driver.

The driver (PC side) computes real-world coordinates of the hands using saved earlier sent from ARCore and pushes them into Monado.

I worked hard on both the Android app and the Monado driver and finally: I have working hand tracking!

It is very laggy, needs many optimizations, and isn’t even in alpha state yet, but I can see that it is possible to do it. This was my dream when I began this project, and finally, it has come true. I hope it will help many people who cannot afford a professional headset :D

So today I finished image settings (you can configure them in the config file). They control how the image is rendered, because every headset has different lenses, distance between lenses, distance from screen to lenses etc. To find your headset’s properties, go to https://wwgc.firebaseapp.com/.

I created new TCP socket logic for configuration sharing and added logic for loading configuration file.

Now I am trying to change the x265enc software encoding to something faster on GPU. Unfortunately, nvenc (nvidia encoding) is really hard to configure properly, as you can see for yourselves.

Phew, I almost went over the time limit (10 hours) :D

I tried to get it running on Windows, and I partially succeeded, but it’s still not stable. Aside from that, I mostly cleaned up the code, improved usability, and removed unnecessary modules from CMake.

Now I’m working on an improved distortion using Monado’s u_distortion_cardboard_calculate (cardboard stands for Google Cardboard btw)

And one more thing: thanks to everyone for the positive feedback on the project — I really didn’t expect that!

Since nobody reviewed my submission, I don’t want to add more features, so I only updated the README and made small changes to the app.

If anyone wants to play VR games on their phone, they can just download the files from Releases and run them without hard compiling!

Devlog 4: I replaced boilerplate LLM code with my own. I also wrote README with instructions, so you can now try it!