okay so omoggle/mogwars was cool but this is cooler:

i noticed that those “backends” only ran on a static frame of your face and only measured landmarks. I decided to make one which bases it off of a 3D scan as well as texture information to estimate skincare.

After some research, I found 3DDFA, a small model which can reconstruct a 3D model of a face from photos.

the app first takes 7 poses by asking you to move your camera and then passes them all into the model. (120x120x3b tensor). The model gives 62 parameters per pose, and then the app takes the median identity-shape parameters across all views -> BFM2009 decoder -> 68 points used to calculate all the structural metrics.

then it uses the texture of the face that’s not in the eyes or mouth and samples them for clarity, eveness, texture, exposure, sharpness, etc. takes the median of that and then calculates a skin value. exact weights for them are found in the code.