#MOTION MATCHING SYSTEM V1 DEVLOG
ok its been a few days and we have gotten a few things done including

• started work on feet and hand bone tracking (the secret sause to motion matching)
• I finished making the large Charuco Board so I can calibrate FreeMoCap.

so its coming along… slowly but that’s ok. the hardest part of all of this is the motion capture though since my setup is really bad (I wish my city had a capture studio somewhere even if it was small)

sheesh its been awhile since I last posted a devlog.

MOTION MATCHING SYSTEM V1 - HIP

Better matching

Change Log/Updates

• Matching changed from world space to local space
• started work on recording motion capture

I got more consistent matching now thanks to changing how it store the data in the dataset and how it matches to it. originally the future trajectory positions where stored as world space values in the dataset and the matcher would match it to the players world space values. if you know how transform systems work in game engines or blender you might know that because of that I had to be in the EXACT same position in the world that the data was recorded (you can see this in my last devlog where I kinda wiggle around in a few areas and get flurries of matches) in which… doesn’t work very well for this because if you had to record your self walking forwards backwards and side to side for ever single value in the game world that would take you FOREVER! so I switched it to local space which is basically where you are in the world based off of something. so like instead of 0, 0, 0 being the root of the world it would be where the character is so the trajectory positions are based around the character transform… do you know what I mean? I hope so because I cant think of a better way to explain it at the moment.

Right now the biggest thing holding me back is the actual motion capture data that I am feeding into this thing because the recording I downloaded… is really bad. so I have been experimenting with a free open sourced software called FreeMoCap which is a marker less triangulation based motion capture software. I will report back when I get some good results out of it (hopefully soon)

TODO

• hand and feet position and velocity matching
• 6 to 8 camera motion capture data
• finish Charuco board for FreeMoCap camera calibration

Video is of the new matching in action the red (and 1 green) points are the players future predicted trajectory the pinkish purple points are the animation fames current trajectory. they should be aligned most of the time

FIRST MATCH!!! WE DID IT! This past week I started working on code to compare and chose a frame to play (the actual matcher) and it works… Kind of? YES IT MACHES… Is it a good match? HECK NO!!!! BUT IT MATCHES!!!! AND THAT JUST MADE 9 MONTHS OF WORK WORTH IT AS I HYPERVENTILATE WATCHING MY MATCHER STUGGLE BADLY LAGING EVERYTHING BUT IT WORKS! now the next step is making it work… properly. the video is of it trying to match. i kinda move around a bit to get it changing data but you can tell when it finds… something because the character will lock onto a frame or start playing the animation.

WE HAVE A WORKING DATASET! Hello hack club! Didja miss me? NO?! WELL TOO BAD!!!!! I now got my motion capture data outputting a dataset with trajectory samples and root positions which is a VERY big step forwards. so as I have mentioned in the past motion matching is literally what MOTION MATCHING is doing. Matching the motion. I have had player predicted trajectory working since before stardance even started but only recently have we started to make progress on the motion capture data that we can MATCH to the players predicted trajectory. the set up I have going on for running the motion data extraction is really cool if I do say so myself. I have a custom editor menu where I can add animation clips and tell it what times in the future I want the trajectory to sampled from then it will compile everything into a scriptable object. plus I also added cool loading bars! anyways we are getting very close to having actual motion matching made from scratch. wow. 9 months this has took so far. its been fun. well… technically not done yet so lets keep up the good work and build the matching algorithm

DataSet Baker V1 Alpha - HIP is in the works!!!!!

Now you might be asking why it’s named HIP, and unless you are from Canada you will most likely not know who I am going to talk about, but the motion matching editor versions are going to be named after Tragically Hip album names (the first one is self titled so HIP).

Anyways, in the last week I have gotten a lot done. I started work on actually figuring out the logic to get the trajectory out of the animation (and I don’t remember if I talked about this before but there were 2 ideas I had. 1 is I put the animation’s root motion through a lot more math to get a proper predicted trajectory just like how the input trajectory works. Or just sample the actual future, but then you would get problems like a 90 degree turn where the animation isn’t even anticipating it yet but the trajectory says it should be.)

I actually got really confused and stuck so I emailed someone. That someone is Daniel Holden, the creator of Learned Motion Matching (and he runs the website where I learned the first input trajectory math).

And WOW!!!! HE RESPONDED:

Hi Asher,

Sorry for the late reply, I have been travelling the last couple of days.

First of all I just wanted to say that it is awesome that you are looking into advanced animation techniques like these already.

(email removed for space) (i might put it in the comments if you want to read it)

I hope that helps!

All the best,

Dan

LIKE HOLY COW!!! I am talking to a LEGEND!!!!

Normal motion matching works as such: you got 2 trajectories, a predicted one from the input then a true trajectory from the animation, then you load all that animation into RAM and use a matcher to play the correct frame.

But what he made was use a neural network instead of the matcher and database WHICH SHRUNK RAM USAGE LIKE CRAZY (and I might have to get working on something like that sometime soon because RAM prices don’t appear to be dropping anytime soon).

Anyways, in the email he said that my second idea was actually right. The idea that I thought was wrong, but he says is just a sad limitation of it and nobody has figured it out yet (yet…).

So that will make life easier because I WILL BE ABLE TO FINISH THE DATASET STUFF HOPEFULLY IN A WEEK!!!!!!!

And to make that dataset I started messing around with Unity’s UI Toolkit and custom editors. So now I have several custom drop downs and then you can open the MOTION MATCHING DATASET EDITOR where you can define all of your animation clips and once I write the rest, be able to press the bake button. It will process every frame and then spit out a Scriptable Object or database.

Now hopefully writing the rest will be easier than writing the UI code because the UI stuff is powerful but not like normal runtime coding.

HELLO STARDANCE!!!!!! I THINK I FIXED IT!!!

Now I am not going to even mention that root motion is working this time because if you have been following along you know that the second “IT WORKS!” comes out of my mouth something breaks.

So last devlog I said I thought it was working but… my character was half way in the floor. after a bit of fiddling around and some google I discovered that the problem was that I had put the root bone in the hips of the character. And apparently in unity that’s a big no no.

So one hour and eight minutes of blender B.S later I Fixed it (hopefully).

I ended up using a blender addon called RootMaker to create a proper root bone at the feet where unity wants it. then I used the addon to transfer all of the root motion data from the hips into the new dedicated root bone. so now hopefully unity will be happy that it has a proper root bone to work with and wont try to shove my character into the earths mantle.

The crazy part of all of this is that I have been doing more of blender and fighting animation during Stardance than actually working on the motion matching so if this continues to work I can get to work on what I am supposed to be doing; working on the actual motion matching.

since I am the music variety of nerd I spend most of my time coding jamming to music and I think I started measuring my programing time in albums rather then hours XD. Over the course of just this weekend I have listened front to back to:

21st Century Breakdown - Green Day
American Idiot - Green Day
Back in Black - AC/DC
Black Holes and Revelations - Muse
The Black Parade - My Chemical Romance

and if you add all of the runtimes of those albums up it matches very close to the time logged in Stardance so it might work XD. anyways I am going to go stretch my legs then maybe play some of The Last of Us Part 2 (which is very appropriate since it too uses motion matching as its animation system)

at this point I should stop saying that I have root motion working because every time I say that I jinx it and something falls apart. so I set everything up wrong (of course) so I need to pull all of the rigs apart once more to fix some errors that I made there and how I set up the root bone (its set at the hips not at the feet). so its been a fun ride (not) plus my laptop keeps trying to update my graphics drivers and every time it crashes unity, blender, plus all of the audio drivers with it. XD. I hope after today I can start work back up on motion matching and not dealing with blender and its shenanigans.

Ok. I may have celebrated a little bit early. but I think I finally got root motion working for real this time. the problem was a combination of blender being blender, transformation offsets, weird scaling and just the names of objects/bones being wrong. though I did get an awesome video from when I was testing and comparing two animations together and root motion sent one of them into the floor so we got a beautiful video of one guy riding the other guys head. anyways lets hope that this keeps working because I just want to get my math mathing and dataset datasetting

This morning I got root motion finally set up in unity. since that now works I can begin working on a data set for the motion matching. I guess I never really explained how motion matching works in any of my other devlogs so I am going to do that right now:

Motion matching is a animating technique where instead of using classic state machines that go if walking play walking animation they go this is your current position, velocity, trajectory, and a lot more and try to MATCH that MOTION to a large database of motion captured animations.
so what I will be working on next is building that database/dataset of animations and the code to pull the current position, velocity, trajectory out of the animations then store each frame in a dataset for the matching code to search though to find one that matches what the character is currently doing on screen. its sounds kinda easy but trust me unless you are a big math/physics person (which I am not) its quite difficult to figure out how it al l works (plus you also need to know animations and blender which SUCKSSSS!)

Also the video is of root motion NOT WORKING!

fighting unity and blender animation systems

I got input trajectory working. The reason it wasn’t before was that movement and the prediction where in different update types (one was in fixed update and one was in regular update) so that works really well now. Now I am setting stuff up to make a animation dataset which involve me modifying my trajectory code to make it work with PRE RECORDED animations

Devlogs From flavortown:
Devlog 1:
Lots of research and trial and error. I have spent all this time looking for an algorithm that works to generate a predicted trajectory from an input. I think I found the right one (I haven’t tested it yet) but if not then I know that I am looking in the right place at Daniel Holden’s articles on spring dampers
check it out here: https://theorangeduck.com/page/spring-roll-call#controllers

Devlog 2:
I got the Trajectory prediction working!!!! after all of that math, research, and pain. it works! I wish I could truly express in words how happy I am. everything is based off of the work of Daniel Holden you can find it here: https://theorangeduck.com/page/spring-roll-call#controllers

Devlog 3:
Fixed jittering issue by adding a lerp to desired trajectory.