One step closer to finally shipping…or so I hope. I speedran shipping this next feature, so it took a lot of time. This feature is mainly for myself. I have an old Mac (running 10.13 from 2017, really old I get itttt). I want my app to run on there, but it currently only supports 10.15+. Hence, I went down a funny rabbit hole in an attempt to add support.

33. Cleaning up dylib signature lookups at run time

One of the key challenges I faced throughout development was how to have a unified codebase for multiple API versions. I decided that instead of version-gating APIs and guessing which beta/normal versions they would be accessible on, I’d test for APIs at run time. I changed VideoToolbox to use this logic: it would attempt to set/use APIs and variables available at higher runtimes, but would work fine without them and have fallbacks. This ensured both legacy and modern systems functioned well.

34. Airplay automations…?

As I was doing competitor research, I noticed many of the other apps supported Mac 10.13-10.15 as well as 10.15+, while mainly only supported 10.15+. Essentially, they started a fake Airplay server, had the Mac connect it, and set the display mode to “Use as separate display”. This would create a virtual display that the app would then use for its own functions. As ScreenExtend requires its own display on older machines, similar to this, where CGVirtualDisplay isn’t an option, I decided to use this method. I had my app automate opening the Airplay menu and connecting as well, in addition to running the fake server.

Most of the process was straightforward, especially due to the abundance of fake Airplay server implementations online and my own experience with apps like LonelyScreen. The one issue of this approach is that only one display can be connected at a time, so any tertiary devices wanting to connect have to wait their turn. I guess the only option is to by a new mac then…

35. Client UI update cleanliness fixes

Due to the one device restriction, I implemented a separate page that would be served if the display slot was being used up. It would be coupled with a short message explaining the consequences of the older Mac version. There were also bugs related to session teardown (keyboard still being locked, video stream still active in the background, etc) that I quickly squashed after finding them.

Soooo in hindsight, I prolly should’ve bumped this up to 0.6.0, but I left it at 0.5.4. Oh well, I’ll increment the version by next release. Only a few more features to go on the feature list before hitting v1 and shipping!!!

Oh my lord, I keep forgetting to write devlogs every 10 hours and am losing hours on each one. Nevertheless, I finally finished audio work! Plus some other things like running claude security agents on the code base and patching stupid vulnerabilities. Here’s how the week went:

28. HiDPI and Mac Issues

Whenever I use my Mac with ScreenExtend, the stream looks blurry due to Macs having HiDPI screens. I fixed this by adding basic HiDPI scaling on Windows. The UI has a DPR slider based on autodetected values. This DPR is multiplied by the screen resolution for a shaper display.

One issue that arose with the new displays was the fact that the display would randomly not scale properly, so the task bar and apps would appear tiny. To mitigate this, I reapplied all display settings after resolution changes and a timeout.

29. AUDIO WORKS!!!!

For the longest time, I tried getting audio to work. On Windows, the process was straightforward: a WASAPI loopback system audio capture session. I wanted to build my own custom driver to capture audio faster, but this would be too expensive to deploy. I read other Rust implementations and quickly wrote the code.

On the other hand, Mac was a whole different issue. On older versions of Mac, system audio recording is not supported. Hence, for Macs below 13.0, I wrote a custom audio driver. I didn’t have a clue about implementing an AudioServerPlugIn on Mac, but worked out the specifics by looking at the Blackhole audio driver. I did have to proxy physical volume buttons to work (i.e. I listen for taps and change the audio level within the driver).

For older Macs, I used ScreenCaptureKit with an extra audio flag. This was a simple fix to capture audio. On even newer Macs, there’s a Process Tap API to record system audio with low latency. Once again, I didn’t have much difficulties implementing these due to the abundance of documentation.

30. Streaming Audio over WebRTC (with A/V sync)

When sending video and audio separately, the two are prone to fall out of sync, so watching movies or videos is difficult. To mitigate these issues, I wrote an A/V sync host clock. This packages the time on the host clock to audio samples and video frames, such that they can be reconstructed and coupled on the receiving end.

On the browser side, I began by implementing a basic audio streamer with an audio element. However, similar to video, there’s always a faster method possible. I used the AudioDecoder class from WebCodecs to implement a worklet class and jitterbuffer. In simple terms, this would decode audio on a separate thread, like WebCodecs prefers, and send the audio back to be played to the user.

31. Windows Video Encoding Speedups

I made a few encoding speedups on Windows targets:

• The NVENC capture encode system uses a texture ring implementation: a pool of 3-5 encoder instances paired with a frame queue, allowing multiple frames to be encoded concurrently
• BGRA encoding is done directly in the library instead of using a third-party implementation
• Unchanged frames in DXGI capture are not encoded again, reducing CPU load

32. Mac Video Encoding Speedups

I made a few encoding speedups on Mac targets:

• Request LatencyCritical activity for the session to ensure App Nap and other sleep measures don’t affect the thread
• Analogous to DXGI, skip encoding when CGDisplayStream reports no dirty rects
• Not an encoding speedup, but preferring 5 GHz for Mac hosted networks to ensure higher speeds

Lotta stuff done this week, am I right? I made lots of progress and finished the bulk of my features, although Linux support is still in the works. I’ll make the 1.0.0 release soon and start a Stardance ship as well. I’ll likely add Linux support after 1.0.0, as I need to research EVDI and the WSL display compositor, as well as other minor tweaks for Windows/Mac. See y’all in the next devlog!

I speedran shipping a buncha features…in the span of a few days. I logged a copious amount of hours on Hackatime building random stuff. I tried some things with recording system audio and building a driver…those didn’t go the best, but I’ll cover that in the next devlog. Here are the main changes this week:

24. Even more quality of life changes

Compared to last time, I wanted the user experience to improve even more. Specifically, for the Windows client, a custom display driver is required. On Macs, the functionality I need is built in, so none is required. However, uninstalling this display driver used to be a pain for users. Now, I bundled it as part of the uninstall script. I wasn’t sure how to do this, but Tauri has many useful guides and issues opened regarding custom WiX/NSIS code.

When the window isn’t focused, join and leave notifications are toasted. This allows the user to stay informed even when the app isn’t open. Basic feature but very useful.

I’d often get annoyed that I needed to keep re-entering the password to log in to a previously connected device. Hence, I added auto-join for devices based on their IP. This ensures that commonly connected devices can save time when reconnecting.

25. Onboarding screens

The app is intuitive to use…or so I thought? Some of my beta testers were confused as to the various features and how to use the interface. Hence, I used NextStepjs to add onboarding components. This would dim the UI and only highlight the component being displayed, along with a message next to it. I liked the styles of NextStepjs a lot, so I chose it. There were some issues regarding window resize, so I made the onboarding exit and reopen to the last step when the window is resized. This way, the positioning of elements was forcefully recalculated.

26. Random security fixes

My app had various vulnerabilities scattered throughout, so I ran a vulnerability Claude skill and patched some holes. The main issues fixed were:

• Checksums of bundled DLLs and resources are checked before being pushed to GitHub Actions
• Tauri CSP and allow list are actually enforced instead of being set to allow all
• TLS cert used for hosting the HTTPS domain has more secure file system permissions to reduce unwanted tampering of files
• Devices are trusted by a uniquely generated token, not by IP, to decrease the chance of impersonators

27. Command-line interface

A majority of the time on this devlog was spent on the CLI. It wasn’t that hard (from a technical perspective), mainly just translating a lot of the UI code into Tauri CLI commands and mapping them correctly in the generated JSON file. One of the harder things to figure out was setting up cross-thread AppState, when the app wasn’t exactly running…turned out I just had to move the initialization command to before the window is created, so it would run inside both CLI and GUI modes. To present the QR codes better. I used qr2term to display the QR codes. By default, all logging is hidden. This turns ScreenExtend into a clean CLI mimicking the GUI. Hoorah!

That’s it for this devlog. Not much to conclude, but plenty more features coming. Stay tuned!

Oh my lord…I crammed a lot of features into this week. I have yet to do proper audits of the codebase as a whole, specifically related to latency optimizations. I added a lot more code that affects the WebRTC pipeline, but I at least got the core features working.

20. Automatic updates

For a while, the app relied on manual updating. It would check the latest version on Github, but users would have to go to the website, download the installer, uninstall the current version, and reinstall the newer version. This also messed up config and was a huge pain. Hence, I generated my own signing keys and used the Tauri updater plugin to automatically update apps. When the app is first launched and if an update is available, it automatically downloads and installs it without any user action.

To make this more robust, I added a progress bar while downloading the update. On Windows specifically, the app must be shut down and restarted, so a blocking modal prevents devices from joining till this is done. I originally made this modal manually, but switched to using AlertDialog from ShadCN.

Another issue I faced was the latest.json manifest. As I used custom filenames when downloading and installing updates, I had to manually patch the JSON. I learned a lot about the Tauri updater Github action through this, as I had to figure out how Tauri generated build filenames and how I could safely update them.

21. Quality of life for client devices

I finished QOL fixes on the client side:

• Remember past devices to quickly import preferences or ban devices (all config saving is IP-based)
• Warn if the user is attempting to connect from the same device (this app is a remote control tool, but you can’t exactly remote control yourself)
• If the screen orientation changes, the virtual display is updated accordingly using CSS media queries to ensure a seamless experience
• A no fullscreen mode was added so users could quickly switch apps and maintain productivity

22. Quality of life for host devices

I converted all in-app modals and error messages to JSON. This allowed me to add i18n language support, so I could easily translate my app to other languages.

To detect and ensure only a single instance of the app was running, I previously used a file lock. However, there was an issue with MSI builds falsely detecting a file lock and not opening. I switched to a basic TCP model for IPC so that multiple processes running can figure out which was there first and shut down the others.

23. More more more Mac features!

I’ve always felt a bit sad for my Mac users. I don’t have a Mac to test on, so I keep borrowing my dad’s Mac, which results in insanely slow development times. Nevertheless, I made some serious progress this week. The biggest thing was a simpler installation process. Many users would complain that they didn’t remember if they’re on Intel or Apple Silicon, so I now distribute a universal DMG.

In terms of the app itself, I added a permissions check for screen recording and input injection. It would run a live test of screen capture or key input to see if the app actually had permissions; therefore, down the road it wouldn’t cause issues. I initially tried to do a soft check without an actual test (i.e., read the settings app), but this didn’t work. The only foolproof way was to call the API itself.

The biggest feature by far was adding Mac input injection. Using accessibility APIs and a charmap similar to Windows, it would inject input onto the Mac screen. I could remotely log in from my Chromebook and get a lot of work done. I mirrored the structure and code on the Windows input injection pipeline, so no major difficulties here.

That’s it for this week! Lotta stuff added, especially for the Mac users. I have a long list of features to continue building, specifically regarding easier Windows uninstall processes, onboarding, and an overall better UX. See you soon!

I had so much fun at the conference in San Francisco! I met a lot of cool people who gave me advice on the app, although I didn’t see the downloads spike 😭. I primarily began working on a SW encoder and other miscellaneous changes:

17. Remote control toggle

Sometimes, the user doesn’t want a client to have remote control access. Although the purpose of the app is to have remote control, it is useful to disable this feature. On the frontend, I added a toggle switch that enables or disables the remote control backend. I was debating whether WebRTC negotiation was required as well, but decided to do so in the end. The remote control used data channels, so an unused data channel would result in slightly lower latency. Renegotiation ensured the stream stayed as efficient as possible without sending redundant data.

18. Update checking

In order for current users to access newer versions, I wanted to add an update functionality. I’m not using Tauri’s update plugin just yet, so I run a simple check. The app, on startup, checks the latest release tag on GitHub. If it’s not equal to the current version (rollback or higher version), the user is prompted to download the newer version. Automatic updates will come next.

19. The big boy: software encoding

This one’s been a long time coming. A lot of machines like VMs or computers with a weird setup just don’t have a working hardware encoder (Intel/AMD/Nvidia), and until now that meant the app straight up didn’t work there. So I built a full CPU-only x264 fallback.

For the pipeline, the frames come as raw BGRA bytes that the other supporters already encoded. They must be converted to I420 before going to libx264, whose build I pulled from OBS. As a side note, the DLL is bundled with the app at runtime to ensure no missing dependencies. As with any encoder, the rate control is ABR with a tight VBV, as well as an infinite GOP. Besides the technical jargon, this just means latency is the north star and nothing else matters.

The whole software variant is added to the encoder vendor enum, and wired into the fallback chain: if NVENC fails, try Intel; if Intel fails, fall back to software before giving up entirely. There’s also a manual override if someone really prefers software encoding, but this is not the default. There’s a big CPU cost compared to hardware encoding, so it’s very much a last resort and only kicks in if absolutely no GPU encoders are available. But “last resort that works” beats “doesn’t work at all.”

That’s it for this week! I plan to have some smaller quality-of-life changes and better updates coming soon…

Sooooo this devlog is mostly comprised of website changes. I have a conference in San Francisco and wanted to polish up the app before publishing. Here are the main changes I made:

13. Random chores and updating the website

I absolutely love shadcn-ui, but so do AI models. In past reviews by stardance raters, as well as family and friends, they pointed out the website looked AI-generated. Hence, I decreased the number of gradients and added basic animations to make the site feel more alive. This was a first for me to implement animations, but the Tailwind docs made it quite simple. I used a specific code-like font instead of the generic fonts I had and tidied up CSS classes. The biggest change was adding a blurred-geometric background for a nicer feel.

14. Redesigning the website completely

I felt it was better to update the website as a whole, so I changed up the user interface. When I tested the app with real users, they complained that they didn’t know how to actually use the app. I changed the hero design to have clearer text and added an SVG-animation component showing how the site actually works. The feature section stayed the same, but there was a highlights grid (inspo https://cider.sh/) to highlight the main features. The FAQ was shortened to my liking, and I pulled some utilities from Tailwind for animations.

In terms of images, I updated the images of the app featured on the website and cleaned up some of the dimensions related to that CSS. The old homepage still lives inside OldHome() but isn’t used anymore.

15. Tidying up the README and build workflows

I edited the README to remove redundant information and make it more developer-focused. I also updated some information regarding GPU builds. For the build workflow, I ensured it would only run if the commit ended in “rebuild”. This ensures unnecessary build minutes are not spent on small changes not requiring a full rebuild.

16. Wake lock API fallback

This app aims to support all devices, but older devices have…funny APIs to say the least. For WebRTC, I implemented the legacy encoded streams API. However, the device would still turn off for seemingly no reason while the video was playing. On older devices, the wake lock API is not supported. However, there’s a unique workaround for this: any video element playing on the webpage will keep the screen from dimming. Hence, I used the NoSleep library to create an off-screen 1-second video element that keeps looping, ensuring the display doesn’t go dark. Some very fun stuff with browsers.

That’s it for this devlog! Mostly minor changes, but the next one will have better SW encoding support among other changes. Stay tuned…

I met with some friends to discuss my app…and realized I gotta lock in with the features. Among the various topics we discussed like the difficult UI or lack of automatic updates, they really wanted remote control features. So, I set out to make the remote display something I can actually touch, click, and type into. And patching some other bugs along the way.

9. Remote input

This is the main headline feature. Streaming pixels one way was solved; now the client had to send input back up the wire and have the host replay it as if it came from real hardware.

Host side (Windows). A new input backend injects events through a synthetic pointer device, so mouse, touch, and pen all seem real as opposed to faked mouse moves. The synthetic device can execute touch or mouse movements from the tablet client. macOS and Linux backends are stubbed and will come later (trust me bro).

The wire. This side had to be optimized for dead-low latency like the rest of the project. Instead of passing strings or using some wrapper library, I made my own binary protocol (a dedicated protocol module) serialized over WebRTC data channels, not the media path. Client-side input.js captures keyboard, mouse, pointer (touch/pen), wheel, clipboard, and drag-drop and ships them across. The code basically catches every single frontend input event and forwards all those events (20+)! The routing is latency-first, same as everything else, with three different paths. fast-path events (pointer moves, wheel) take an unreliable channel, where a dropped packet is cheaper than a late one, while clicks, keys, and clipboard take the reliable channel where every event must land exactly once. It’s not that big of a deal if one small mouse delta doesn’t go through, but it definitely is a big deal if a click doesn’t go through.

10. Browser compatibility detections

Not every browser can do WebCodecs, secure contexts, or the input paths above. Rather than fail silently, I added a compatibility check on both the browser and client up front. It shows compatibility modals with per-feature lists to ensure the user knows what’s going on.

11. Some more small fixes

I added two quality-of-life additions: avatars (upload custom profile pictures) and zoom controls (ctrl/cmd +/-).

12. Encoders that don’t explode when funny stuff happens

Oh my god…one of the biggest issues by far. When I ran the NVIDIA encoder at high FPS, it didn’t crash. However, the Intel Quick Sync encoder would arbitrarily crash at high FPS and wouldn’t give a reason why. I audited a buncha different parts of my code and eventually found that FPS was the issue, not resolution or low-latency flags.

Hence, I added a binary search that probes the hardware for its true maximum supported fps and clamps to it before the encoder can choke. Essentially, I ask the encoder “can you do 1920x1080 at 240 fps?” –> “no” –> “what about 120 fps?” –> “yes” –> “what about 180 fps?”. This keeps repeating till the max FPS is found and streamed to the client.

Rounding out the release, I made a variety of different UI fixes like the modal fixes, backdrop overlap, a stray uninstall-modal backdrop, and cloud reconfig errors. More progress to come!

ScreenExtend Devlog: Building the Rust Engine Under a Wireless Second Monitor

TL;DR

ScreenExtend turns any device with a browser into a real, wireless extended monitor for a Windows host, no client app and no cables (it began so my brother could use his iPad for notes at debate tournaments). Scan a QR code and the host spins up a real virtual display, streamed over WebRTC with hardware H.264. The React/TS control UI existed before; everything else (Rust core, GPU pipeline, WebCodecs client, cloud relay) was built from scratch, always latency-first, quality-second.

1. The spine: Tauri + a typed Rust↔TS bridge

tauri-specta exports a fully-typed src/lib/bindings.ts for commands (TS→Rust) plus typed events (Rust→TS): devices, network, cloud status, logs. A code-generated bridge beats maintaining one by hand. Let the code write itself!

2. Virtual displays: faking a real monitor

Each client gets its own virtual display the OS treats as real hardware, a true second screen, not mirroring. On join the host creates one sized to the client, forces extend topology, then captures it once attached. On Windows a signed IDD driver installs at runtime (via nefconc + certutil), and it tears down cleanly on leave.

3. First light: WebRTC + WHEP + H.264

The media path is WebRTC (real-time, in every browser). Signaling is WHEP: the client POSTs an SDP offer, the host answers in-response, no signaling server. Codec is H.264, universal hardware decode on clients and encode (NVENC/QSV/AMF) on hosts. The first release streamed NVENC to a plain <video>; it worked but was laggy. It binds HTTP + HTTPS (self-signed via rcgen), since WebCodecs needs a secure context.

4. The pipeline: capture → encode → broadcast

Capture is Windows Graphics Capture (later a custom fork; DXGI Desktop Duplication fallback), with a repeater thread for idle keepalive/IDRs and transient-stall tolerance.

5. Going vendor-specific: the part that made it fast

Generic capture→CPU-copy→encode is slow everywhere; the win: keep the frame on the GPU from capture to bitstream. Per vendor:

• NVENC (NVIDIA): no SDK linked (API dynamically loaded); zero-copy writes capture into a shared D3D11 texture behind a keyed mutex, else a CPU bridge.
• Intel QSV/oneVPL: same-adapter capture+encode fuses the downscale into the VPP pass (BGRA→NV12 + resize in one shot).
• macOS (VideoToolbox): ScreenCaptureKit (12.3+, CGDisplayStream fallback) via objc2 runtime interop only, so one binary loads on every macOS version, same IOSurface zero-copy.

6. The client: WebCodecs over <video>

Fast path: an RTCRtpScriptTransform feeds a worker VideoDecoder (optimizeForLatency: true) onto an OffscreenCanvas, latency hints zeroed. A <video> fallback covers missing WebCodecs or stalls. It also feels like a monitor: fullscreen + input/wake locks, /leave beacon on exit.

7. Adaptive bitrate, sessions & security

A BWE driver reads getStats every 250ms, pushing a smoothed bitrate into the encoder. Changing settings triggers in-place renegotiation (a reconfig/kick epoch the client polls) with no display teardown. Each host is gated by a session ID + 6-digit OTP (OtpLimiter lockout); every device is isolated, over HTTPS + DTLS/SRTP.

8. Networking + status (v0.2.3)

Same-network is direct, no servers. Offline mode stands up an ad-hoc hosted network, no infrastructure needed. Cross-network adds a cloud relay (WebSocket to session.screenextend.app) tunneling signaling while media stays P2P; a self-hosted TURN server covers hard NATs. A logbus feeds an in-app terminal. Working today: Windows/macOS hosts, per-vendor GPU encode, per-device displays, adaptive bitrate, OTP sessions, offline mode, and cloud relay.