FormCheck

Devlog 17:
I completed the apk build and published FormCheck v1.0 on GitHub. I also added a Readme. I tested the apk on my emulator and did all finishing touches. This is my last devlog before I ship this project.

Devlog 16:
Performed some final checks and implemented some final features. At first I was adding some features, then I noticed everything was repetitive. Because of this I handed some of the tasks to Codex so it could complete the repetitive tasks. It just added some more data to the results screen, a cancel button, and rotation support. I added most of the details on the results page.

Devlog 15:
I moved the app’s results onto a separate page so everything is cleaner. The new screen displays feedback, if a repetition was detected, and the timestamps. I check both a valid squat and an unsupported video to verify error messages. This took a long time because I had to learn how to use intents, and also how to use scrollable areas. I am finishing up the app, and almost ready to ship.

Devlog 14:
I started preparing for the feedback. I am planning to mostly give feedback on timing/tempos, so I started adding some functions for that. The first thing I did was correct the scaling of video coordinates before calculating knee angles to ensure higher accuracy. Then I added a lowering-time estimator that looks for sustained movement instead of just angles at timestamps. I ensured to account for rectangular frames, leg switches, observation gaps, and timing integration. All tests passed. The next steps are actually providing feedback from this data, and then connecting this to the UI.

Devlog 13:
This devlog was mostly some final touches. I did add 3 things, though. The first of which is a frame progress display on the UI. This shows the user of many of the total frames have been processed after they start the analysis. Second, I added some text that was a disclaimer on the limitations of the analysis. The third thing I added was a reset between long gaps. So if the squat takes too long to get to the next state, then the detection resets and it starts looking for another starting squat. The first test I did was that I sampled different repetition intervals to determine which had the most detail but didn’t take too long to process. I landed on sampling frames every 250 Ms. Then I tested all the different UI states on the emulator using different videos. I then added tests for all the new features that I just mentioned to you. The next step is probably going to be actually implementing feedback.

Devlog 12:
I added some more meaningful upgrades this time. I added functionality so that the app could detect a real squat repetition. This works by combining this with the pose sequence analyzer, which contains poses and their timestamps. By mapping the angle formed by the hip, knee, and ankle at different intervals of the video, the app is able to detect if the angle got gradually lower, then gradually got higher. If these angles pass the thresholds needed to create a straight leg angle and a squatted down angle, then we can conclude that a squat repetition was performed. To accomplish this I added many different smaller functions, then combined them all together to get the final result. This allowed me to test each part individually to ensure they all worked. I also added this to the UI of the app, which you can see in the image. Finally I added real video validation, which basically means that I tested the current state of the app with a real video of a person doing a squat. The app was able to successfully recognize the repetition of the squat.

Devlog 11:
Now I added a few things. First I added a function that can sample frames at different intervals and stores the MediaPipe result with the timestamp of the frame. I also added hip/knee/ankle extraction for both sides and also confidence-based leg section. Codex wrote a few methods (2-3), and I wrote the rest. I then added timestamped knee angle measurements, and also connected the leg selector to the angle calculator. Finally, I added a sequence analyzer that keeps the valid measurements and skips the unclear frames. I added test for all these functions. There is a little more than this, but this is the gist of what I have been working on right now.

Devlog 10:
This time was some meaningful upgrades. First of all I added a function to extract a bitmap from a video given a timestamp. Then I added a separate function in order to convert that bitmap into an MPImage. This allows me to access it from anywhere. Finally I created a new class that combines the decoding, conversion, and pose processing tasks into one pipeline. I also added tests for all of these. Currently only one frame processes, but soon I will start to improve this.

Devlog 09:
Now the app stores the selected video’s Uri in an activity field, meaning it can be accessed for later processing instead of only the status text. Also frame timestamps can now be generated. This means that you can input the video duration as well as the interval that you want the timestamps to be generated, then you will get a set of timestamps for the video. The video metadata can also now be read. It uses Android’s MediaMetadataRetriever to open the video. Currently only the duration of the video can be retrieved, but later you will be able to decode individual frames. Finally I added tests for all of these. For the metadata retriever specifically I uploaded a test video to confirm that I could get an accurate measurement. Also updated the documentation.

Devlog 08:

I finally added the MediaPipe dependency, as well as a few functions. First of all I also added the Pose Landmarker Full model for it to actually detect poses. Then I created a manager function to initialize and close MediaPipe. I then added a function that could process an already decoded video frame with a timestamp using MediaPipe. I added tests for all of these to ensure they work when the app runs, and updated all the logs.

Devlog 07:

I changed the emulator from being on device to remote via Firebase. This allows me to test the app cleanly without lagging my computer. The only problem is I get a limited 30 minutes per month, so I have to use them wisely. I also updated the Point2D function to reject non-finite points. This prevents errors on the future. I also added a class that combines all the other functionality into one. It returns an OptionalDouble, which is useful when you don’t know if you are going to get a result. This new function accepts three LandmarkObservations, Then if the result is valid, it returns the angle degrees between the points. If not, it returns nothing.

Devlog 06:

I added the landmark quality gate, which helps determine whether the visibility and presence of the point meet the minimum requirements. If it does, then it passes. If it doesn’t, then the point fails. From here there are two options, either the app tries to find another point, or the app says it cannot reliably determine your form from that video

I also added the AI use log. I mainly did this because I will be submitting this to the Congressional App Challenge.

Devlog 05:

Today I added landmark storage and reliability tests. What this is is basically a new object that can store a point, its visibility, and presence. The point is the point itself, the visibility says how well the object can be seen, and the presence determines how much of the object is actually in the video frame. These three factors help determine how well the video can be analyzed. If the video cannot be analyzed well enough, then FormCheck will send a message saying so (This hasn’t been implemented yet, but this is the basic idea of why this is there in the first place).

Devlog 04:

I added functions for measuring the angle based on three points (the joints). Basically how this works is that it first takes in three points, the first endpoint, a vertex, and a second endpoint. Then it uses the dot product formula with the two vectors to find the angle between the two lines.

Devlog 03:

I added the file picking dialogue. Now whenever you click the “Select a video” button the android folder picker opens and you can choose a video. I think next is working on the actual pose detection logic.

Devlog 02:

I added the initial video selection interface. I added some text to describe the app, and a button for the user to be able to select a video to upload. Currently, the button does not do anything. Next I will have to continue working on the button and the actual pose estimation part of this.

Devlog 01:

This is the first devlog of the FormCheck project, an Android app that helps you check your form during workouts. Currently I just set everything up in Android Studio, connected Hackatime, and set up GitHub.