Oh my lord, I keep forgetting to write devlogs every 10 hours and am losing hours on each one. Nevertheless, I finally finished audio work! Plus some other things like running claude security agents on the code base and patching stupid vulnerabilities. Here’s how the week went:

---

28. HiDPI and Mac Issues

Whenever I use my Mac with ScreenExtend, the stream looks blurry due to Macs having HiDPI screens. I fixed this by adding basic HiDPI scaling on Windows. The UI has a DPR slider based on autodetected values. This DPR is multiplied by the screen resolution for a shaper display.

One issue that arose with the new displays was the fact that the display would randomly not scale properly, so the task bar and apps would appear tiny. To mitigate this, I reapplied all display settings after resolution changes and a timeout.

---

29. AUDIO WORKS!!!!

For the longest time, I tried getting audio to work. On Windows, the process was straightforward: a WASAPI loopback system audio capture session. I wanted to build my own custom driver to capture audio faster, but this would be too expensive to deploy. I read other Rust implementations and quickly wrote the code.

On the other hand, Mac was a whole different issue. On older versions of Mac, system audio recording is not supported. Hence, for Macs below 13.0, I wrote a custom audio driver. I didn’t have a clue about implementing an AudioServerPlugIn on Mac, but worked out the specifics by looking at the Blackhole audio driver. I did have to proxy physical volume buttons to work (i.e. I listen for taps and change the audio level within the driver).

For older Macs, I used ScreenCaptureKit with an extra audio flag. This was a simple fix to capture audio. On even newer Macs, there’s a Process Tap API to record system audio with low latency. Once again, I didn’t have much difficulties implementing these due to the abundance of documentation.

---

30. Streaming Audio over WebRTC (with A/V sync)

When sending video and audio separately, the two are prone to fall out of sync, so watching movies or videos is difficult. To mitigate these issues, I wrote an A/V sync host clock. This packages the time on the host clock to audio samples and video frames, such that they can be reconstructed and coupled on the receiving end.

On the browser side, I began by implementing a basic audio streamer with an audio element. However, similar to video, there’s always a faster method possible. I used the AudioDecoder class from WebCodecs to implement a worklet class and jitterbuffer. In simple terms, this would decode audio on a separate thread, like WebCodecs prefers, and send the audio back to be played to the user.

---

31. Windows Video Encoding Speedups

I made a few encoding speedups on Windows targets:

• The NVENC capture encode system uses a texture ring implementation: a pool of 3-5 encoder instances paired with a frame queue, allowing multiple frames to be encoded concurrently
• BGRA encoding is done directly in the library instead of using a third-party implementation
• Unchanged frames in DXGI capture are not encoded again, reducing CPU load

---

32. Mac Video Encoding Speedups

I made a few encoding speedups on Mac targets:

• Request LatencyCritical activity for the session to ensure App Nap and other sleep measures don’t affect the thread
• Analogous to DXGI, skip encoding when CGDisplayStream reports no dirty rects
• Not an encoding speedup, but preferring 5 GHz for Mac hosted networks to ensure higher speeds

---

Lotta stuff done this week, am I right? I made lots of progress and finished the bulk of my features, although Linux support is still in the works. I’ll make the 1.0.0 release soon and start a Stardance ship as well. I’ll likely add Linux support after 1.0.0, as I need to research EVDI and the WSL display compositor, as well as other minor tweaks for Windows/Mac. See y’all in the next devlog!