Devlog #2: How I fought FFmpeg and won. 🥊

Last time, I successfully built an audio pipeline that listens through a 45 minute match recording and spits out the exact timestamps of every major spicy moment. (Through crowd volume analysis.)

My goal for today: Pass those timestamps into FFmpeg, slice those highlights out, and save the video. Easy, right? Right?

The Final Boss

And then I saw him: NAL Unit Errors. 

FFmpeg immediately started throwing a fit. All I could stare at was a wall of red text for hours on end, trying to figure out what was going on. The error seemed to occur when I tried to slice a video after the 5 minute mark.

I spent hours tweaking my FFmpeg commands. I tried re-encoding, I tried Apple Silicon hardware acceleration h264_videotoolbox, but nothing worked. It always crashed at exactly the 300-second mark.

The Plot Twist

The consistent crash-time gave it away. Turns out, the video had a corrupted index map (the moov atom) due to me downloading it from a sketchy website. Bruh.

 When FFmpeg tried to “fast-seek” (-ss), the map sent it to the wrong byte address, and landed in the middle of some garbage data stream, causing all the errors. ⚠️

The Solution

And so I switched to yt-dlp to ensure download quality. Then, I sliced each individual highlight moment using this command:

ffmpeg -y -ss <clip_start_time> -i match.mp4  -t <duration> -c:v copy -c:a copy temp_clip_i.mp4

And then, after we got all the individual clips, I could easily concatenate them using the following cmd: 

ffmpeg -y -f concat -safe 0 -i clips_list.txt -c copy highlights.mp4

And with that, we get our first demo of this project working quite well! There’s some fine tuning to do with the parameters like continuity_time and sustained_time, but it’s a work in progress. 🚧

Next Up

Now I need to integrate all the different aspects of this project together. The goal is to be able to provide the YT link of the video you want to generate highlights of, and the program being able to automatically download, process, and generate an accurate highlight montage of the game, with all the goals and important moments included in it. See you later!

Take a look at the processed highlights video demo below! As always, please do drop your feedback in the comments section. Took about 18 seconds to process 45 minutes of match recording

It’s World Cup season, hurray!!!

Having to sit through 90 minutes to create a highlight video is painful (trust me, I’ve tried). But that got me thinking: What if you could automatically generate a highlight video, given the full match recording?

I booted up my laptop and put on my programming hat.

I needed a way to identify the important “highlight-worthy” moments in a game, and three approaches immediately came into mind:

Approach #1: Crowd Noise Analysis

Initially, I encountered so many problems dealing with crowd noise. What do you MEAN Barcelona fans are louder than Man City fans? Dealing with variable crowd noise baselines was a real headache.

But then, I struck gold! By calculating the RMS (Root Mean Square) crowd energy, and considering the top 10% percentile as big moments, my highlight clipper could dynamically adapt to different matches, no matter the baseline volume.

All the peaks refer to pivotal moments, which is exactly what I wanted! This approach certainly looks promising, and it identified all the key goals, big chances and fouls that happened in the first half of Barcelona vs Real Madrid 2026.

Approach #2: Whistle Detection

I really wanted this method to work, but I’m sorry to inform you that I could not get this to function reliably. There’s plenty of other noises that the crowd makes that fall in the 2-4 kHz range.

I tried plotting a spectogram, but that only left me even more confused. Let’s hope I can make sense of it by next time!

I hope you’ll follow my journey, and please drop a like if you thought this was interesting!