sheesh I’ve done a bit in the last week.

So we have

• added the ability to match bones in the matcher
• got the matcher to handle multiple animation clips

and I guess that’s it… I honestly thought I did more… on paper it doesn’t actually seem like that much.. huh. OH WELL! I also found a MASIVE collection of animation from a unreal engine demo project so I’m probably going to extract that and retarget TONS of that over the next few days because imma say it the problem with the matcher is the lack of good data at this point.

Pink/Purple - Animation frame data position
Red/green - predicted trajectory

OK OK OK

IT WORKS!!!!!

so we got bone positions to work. I’m done for the day. anyways I literally just had to flip my subtraction around… yes… its stupid…. anyways went from this: RootPositions[i][j] - BoneTransform.position to this: BoneTransform.position - RootPositions[i][j]. ok mic drop… I did say I would ship by the end of the month of august and clearly that didn’t happen but it will (hopefully) be this week

Pink sphere is the foot pos from data set
Blue is the other models foot pos not from dataset
then red is root position

(Also 2 devlogs in one day! I’m on fire!)

IM SO CLOSE!!!!!!!!!

LIKE LATER TODAY (maybe can’t promise anything or the programing gods will kill me) I WILL HAVE MOTION MATCHING IN A GOOD DEMOABLE STATE FOR ME TO SHIP! I have got the bone data to work and actually spit out something useful by changing my whole approach and instead of sampling curves (which could work I just gotta deal with all the weird problems) I have created a unity scene in memory it spawns in the rig samples all the bone transform.positrons making unity do all the math and hard work before it returns me world positions of all of the bones I told it to track. the only problem I am having now is that I got to convert those world positions back into local position so that they can work anywhere (and yes I just converted them from local position to world position and this might sound counter intuitive but I need to change the parent of the local position) I remember this being easy for trajectory matching but maybe since this is WAYY more sensitive to rotation I need to try a different approach. the final stretch is approaching us dear hack clubbers. I got the rest of today then school starts tomorrow.

LETS! LOCK! IN! YEAAAAAAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH!!!!

btw the pink sphere is the foot potion… or what should be the foot position. Blue is the actual foot position. Red is root Position

Note to self… don’t freehand code… plan everything out on paper because I’m just experiencing Murphey’s law at this point :-<

so…. the data that I didn’t know if it was right or not was incorrect… VERY incorrect (kinda) so that data was in local space I need it in world space but to convert it you either need to do the conversion through a transform (which requires a transform to reference) or do what I am doing (since I don’t have a easily accessible transform (unless I’m stupid and don’t know of one)) so I am going up the paths and for each object extracting their position and transforms (in local position) then adding it up until I eventually hit the root which would be in world space so now its converted (I’m doing the same for rotation too with more math) anyways my code for this entire project was never pre planned and I never normally pre plan but this is one of the times where once it works I’m going to figure out how it works (because who am I kidding I don’t really know) then I’m writing all of it down on paper and optimizing the loops and everything before jumping back into the editor to re-write the bake() function from scratch leading to MUCH BETTER READABILTY and hopefully speed because it takes a good 5 seconds from my for loops and brute force. anyways if you cant tell this is VERY BORING WORK and I really just want it to work SO BADLY because once I get everything in world position the next steps should be really easy but I shouldn’t say that because at this point if I do I might jinx myself

WOW JUST WOW!!! So I toured the Vicon Set-up at my local univercity and it was amazing. I loved every moment of it. they even put markers on me and captured some data (not any I can use for motion matching but still WOW THAT TECH IS COOL)

In more on topic news I got the dataset maker to output bone data into the database. is it the correct data… idk but I will figure that out!

(Photo of me standing forward is just a general photo with the markers. photo of me looking away from camera with hand in air is me showing off all the retroreflective markers (and EMGs) then there is a photo of the compiled dataset)

THERE WE GO! finally we can select what bones we want to track from a nested dropdown. now I just need to implement it into the baking logic so I can get a finished dataset out of dataset maker. perfect timing too. I have gotten a few emails back from the university and they are willing to give me a tour of the Vicon system and there may be a chance one day to capture data on it so hopefully everything works out and I can capture on the same tech that AAA game studios use which would be a dream come true and would blow my mind.

There we go. I think I finally got it working. JEEZ UNITY NEEDS BETTER DOCUMENTATION ON THE UI TOOLKIT LIKE WOW ITS THAT BAD…. anyway I got the custom lists working now I just gotta add the actual bone selections then write in the logic to add everything to the datasets. if you look at the photo that’s roughly how it will look to use the dataset maker. not sure if I mentioned this in the last devlog but I finally messaged my local university to see if they will let me either tour or use their vicon system so I’m hoping something comes of that because that would be really cool. anyways its really late I pulled another programmer and stayed up till midnight working. see yall later.

Pretty boring stuff happening right now. I have been working to implement bone positional and velocity tracking into the dataset builder but since I wanted to use a custom class to do it the unity UI Toolkit decided to pick a fight and be really annoying to do it in. if anyone wants to write a custom class in unity and have it show up in a list view use this: link I also am working on processing some mocap data but its not going so well right now and I am getting pretty tired with dealing with these markerless systems so we will see if my local university will let me use or even tour their 10 cam Vicon setup (cross your fingers for me) anyways I still have one mocap session to prosses so I will probably do that tomorrow since I can just let it run while I am doing other things. also I just watched these videos where this guy is building open sourced optical mocap cameras and I think I might just funnel all my hack club hours into pcb credits to build a few of them.
part 1 vid: https://youtu.be/kYVqL_DqBis?si=CPKxKiLa8cxf3M1t

Part 2 vid: https://youtu.be/WJ13C_1hDIA?si=hMlQ8u5ofEBf-cFS

#MOTION MATCHING SYSTEM V1 DEVLOG
ok its been a few days and we have gotten a few things done including

• started work on feet and hand bone tracking (the secret sause to motion matching)
• I finished making the large Charuco Board so I can calibrate FreeMoCap.

so its coming along… slowly but that’s ok. the hardest part of all of this is the motion capture though since my setup is really bad (I wish my city had a capture studio somewhere even if it was small)

sheesh its been awhile since I last posted a devlog.

MOTION MATCHING SYSTEM V1 - HIP

Better matching

Change Log/Updates

• Matching changed from world space to local space
• started work on recording motion capture

I got more consistent matching now thanks to changing how it store the data in the dataset and how it matches to it. originally the future trajectory positions where stored as world space values in the dataset and the matcher would match it to the players world space values. if you know how transform systems work in game engines or blender you might know that because of that I had to be in the EXACT same position in the world that the data was recorded (you can see this in my last devlog where I kinda wiggle around in a few areas and get flurries of matches) in which… doesn’t work very well for this because if you had to record your self walking forwards backwards and side to side for ever single value in the game world that would take you FOREVER! so I switched it to local space which is basically where you are in the world based off of something. so like instead of 0, 0, 0 being the root of the world it would be where the character is so the trajectory positions are based around the character transform… do you know what I mean? I hope so because I cant think of a better way to explain it at the moment.

Right now the biggest thing holding me back is the actual motion capture data that I am feeding into this thing because the recording I downloaded… is really bad. so I have been experimenting with a free open sourced software called FreeMoCap which is a marker less triangulation based motion capture software. I will report back when I get some good results out of it (hopefully soon)

TODO

• hand and feet position and velocity matching
• 6 to 8 camera motion capture data
• finish Charuco board for FreeMoCap camera calibration

Video is of the new matching in action the red (and 1 green) points are the players future predicted trajectory the pinkish purple points are the animation fames current trajectory. they should be aligned most of the time

FIRST MATCH!!! WE DID IT! This past week I started working on code to compare and chose a frame to play (the actual matcher) and it works… Kind of? YES IT MACHES… Is it a good match? HECK NO!!!! BUT IT MATCHES!!!! AND THAT JUST MADE 9 MONTHS OF WORK WORTH IT AS I HYPERVENTILATE WATCHING MY MATCHER STUGGLE BADLY LAGING EVERYTHING BUT IT WORKS! now the next step is making it work… properly. the video is of it trying to match. i kinda move around a bit to get it changing data but you can tell when it finds… something because the character will lock onto a frame or start playing the animation.

WE HAVE A WORKING DATASET! Hello hack club! Didja miss me? NO?! WELL TOO BAD!!!!! I now got my motion capture data outputting a dataset with trajectory samples and root positions which is a VERY big step forwards. so as I have mentioned in the past motion matching is literally what MOTION MATCHING is doing. Matching the motion. I have had player predicted trajectory working since before stardance even started but only recently have we started to make progress on the motion capture data that we can MATCH to the players predicted trajectory. the set up I have going on for running the motion data extraction is really cool if I do say so myself. I have a custom editor menu where I can add animation clips and tell it what times in the future I want the trajectory to sampled from then it will compile everything into a scriptable object. plus I also added cool loading bars! anyways we are getting very close to having actual motion matching made from scratch. wow. 9 months this has took so far. its been fun. well… technically not done yet so lets keep up the good work and build the matching algorithm

DataSet Baker V1 Alpha - HIP is in the works!!!!!

Now you might be asking why it’s named HIP, and unless you are from Canada you will most likely not know who I am going to talk about, but the motion matching editor versions are going to be named after Tragically Hip album names (the first one is self titled so HIP).

Anyways, in the last week I have gotten a lot done. I started work on actually figuring out the logic to get the trajectory out of the animation (and I don’t remember if I talked about this before but there were 2 ideas I had. 1 is I put the animation’s root motion through a lot more math to get a proper predicted trajectory just like how the input trajectory works. Or just sample the actual future, but then you would get problems like a 90 degree turn where the animation isn’t even anticipating it yet but the trajectory says it should be.)

I actually got really confused and stuck so I emailed someone. That someone is Daniel Holden, the creator of Learned Motion Matching (and he runs the website where I learned the first input trajectory math).

And WOW!!!! HE RESPONDED:

Hi Asher,

Sorry for the late reply, I have been travelling the last couple of days.

First of all I just wanted to say that it is awesome that you are looking into advanced animation techniques like these already.

(email removed for space) (i might put it in the comments if you want to read it)

I hope that helps!

All the best,

Dan

LIKE HOLY COW!!! I am talking to a LEGEND!!!!

Normal motion matching works as such: you got 2 trajectories, a predicted one from the input then a true trajectory from the animation, then you load all that animation into RAM and use a matcher to play the correct frame.

But what he made was use a neural network instead of the matcher and database WHICH SHRUNK RAM USAGE LIKE CRAZY (and I might have to get working on something like that sometime soon because RAM prices don’t appear to be dropping anytime soon).

Anyways, in the email he said that my second idea was actually right. The idea that I thought was wrong, but he says is just a sad limitation of it and nobody has figured it out yet (yet…).

So that will make life easier because I WILL BE ABLE TO FINISH THE DATASET STUFF HOPEFULLY IN A WEEK!!!!!!!

And to make that dataset I started messing around with Unity’s UI Toolkit and custom editors. So now I have several custom drop downs and then you can open the MOTION MATCHING DATASET EDITOR where you can define all of your animation clips and once I write the rest, be able to press the bake button. It will process every frame and then spit out a Scriptable Object or database.

Now hopefully writing the rest will be easier than writing the UI code because the UI stuff is powerful but not like normal runtime coding.

HELLO STARDANCE!!!!!! I THINK I FIXED IT!!!

Now I am not going to even mention that root motion is working this time because if you have been following along you know that the second “IT WORKS!” comes out of my mouth something breaks.

So last devlog I said I thought it was working but… my character was half way in the floor. after a bit of fiddling around and some google I discovered that the problem was that I had put the root bone in the hips of the character. And apparently in unity that’s a big no no.

So one hour and eight minutes of blender B.S later I Fixed it (hopefully).

I ended up using a blender addon called RootMaker to create a proper root bone at the feet where unity wants it. then I used the addon to transfer all of the root motion data from the hips into the new dedicated root bone. so now hopefully unity will be happy that it has a proper root bone to work with and wont try to shove my character into the earths mantle.

The crazy part of all of this is that I have been doing more of blender and fighting animation during Stardance than actually working on the motion matching so if this continues to work I can get to work on what I am supposed to be doing; working on the actual motion matching.

since I am the music variety of nerd I spend most of my time coding jamming to music and I think I started measuring my programing time in albums rather then hours XD. Over the course of just this weekend I have listened front to back to:

21st Century Breakdown - Green Day
American Idiot - Green Day
Back in Black - AC/DC
Black Holes and Revelations - Muse
The Black Parade - My Chemical Romance

and if you add all of the runtimes of those albums up it matches very close to the time logged in Stardance so it might work XD. anyways I am going to go stretch my legs then maybe play some of The Last of Us Part 2 (which is very appropriate since it too uses motion matching as its animation system)

at this point I should stop saying that I have root motion working because every time I say that I jinx it and something falls apart. so I set everything up wrong (of course) so I need to pull all of the rigs apart once more to fix some errors that I made there and how I set up the root bone (its set at the hips not at the feet). so its been a fun ride (not) plus my laptop keeps trying to update my graphics drivers and every time it crashes unity, blender, plus all of the audio drivers with it. XD. I hope after today I can start work back up on motion matching and not dealing with blender and its shenanigans.

Ok. I may have celebrated a little bit early. but I think I finally got root motion working for real this time. the problem was a combination of blender being blender, transformation offsets, weird scaling and just the names of objects/bones being wrong. though I did get an awesome video from when I was testing and comparing two animations together and root motion sent one of them into the floor so we got a beautiful video of one guy riding the other guys head. anyways lets hope that this keeps working because I just want to get my math mathing and dataset datasetting

This morning I got root motion finally set up in unity. since that now works I can begin working on a data set for the motion matching. I guess I never really explained how motion matching works in any of my other devlogs so I am going to do that right now:

Motion matching is a animating technique where instead of using classic state machines that go if walking play walking animation they go this is your current position, velocity, trajectory, and a lot more and try to MATCH that MOTION to a large database of motion captured animations.
so what I will be working on next is building that database/dataset of animations and the code to pull the current position, velocity, trajectory out of the animations then store each frame in a dataset for the matching code to search though to find one that matches what the character is currently doing on screen. its sounds kinda easy but trust me unless you are a big math/physics person (which I am not) its quite difficult to figure out how it al l works (plus you also need to know animations and blender which SUCKSSSS!)

Also the video is of root motion NOT WORKING!

fighting unity and blender animation systems

I got input trajectory working. The reason it wasn’t before was that movement and the prediction where in different update types (one was in fixed update and one was in regular update) so that works really well now. Now I am setting stuff up to make a animation dataset which involve me modifying my trajectory code to make it work with PRE RECORDED animations

Devlogs From flavortown:
Devlog 1:
Lots of research and trial and error. I have spent all this time looking for an algorithm that works to generate a predicted trajectory from an input. I think I found the right one (I haven’t tested it yet) but if not then I know that I am looking in the right place at Daniel Holden’s articles on spring dampers
check it out here: https://theorangeduck.com/page/spring-roll-call#controllers

Devlog 2:
I got the Trajectory prediction working!!!! after all of that math, research, and pain. it works! I wish I could truly express in words how happy I am. everything is based off of the work of Daniel Holden you can find it here: https://theorangeduck.com/page/spring-roll-call#controllers

Devlog 3:
Fixed jittering issue by adding a lerp to desired trajectory.