I had so much fun at the conference in San Francisco! I met a lot of cool people who gave me advice on the app, although I didn’t see the downloads spike 😭. I primarily began working on a SW encoder and other miscellaneous changes:

17. Remote control toggle

Sometimes, the user doesn’t want a client to have remote control access. Although the purpose of the app is to have remote control, it is useful to disable this feature. On the frontend, I added a toggle switch that enables or disables the remote control backend. I was debating whether WebRTC negotiation was required as well, but decided to do so in the end. The remote control used data channels, so an unused data channel would result in slightly lower latency. Renegotiation ensured the stream stayed as efficient as possible without sending redundant data.

18. Update checking

In order for current users to access newer versions, I wanted to add an update functionality. I’m not using Tauri’s update plugin just yet, so I run a simple check. The app, on startup, checks the latest release tag on GitHub. If it’s not equal to the current version (rollback or higher version), the user is prompted to download the newer version. Automatic updates will come next.

19. The big boy: software encoding

This one’s been a long time coming. A lot of machines like VMs or computers with a weird setup just don’t have a working hardware encoder (Intel/AMD/Nvidia), and until now that meant the app straight up didn’t work there. So I built a full CPU-only x264 fallback.

For the pipeline, the frames come as raw BGRA bytes that the other supporters already encoded. They must be converted to I420 before going to libx264, whose build I pulled from OBS. As a side note, the DLL is bundled with the app at runtime to ensure no missing dependencies. As with any encoder, the rate control is ABR with a tight VBV, as well as an infinite GOP. Besides the technical jargon, this just means latency is the north star and nothing else matters.

The whole software variant is added to the encoder vendor enum, and wired into the fallback chain: if NVENC fails, try Intel; if Intel fails, fall back to software before giving up entirely. There’s also a manual override if someone really prefers software encoding, but this is not the default. There’s a big CPU cost compared to hardware encoding, so it’s very much a last resort and only kicks in if absolutely no GPU encoders are available. But “last resort that works” beats “doesn’t work at all.”

That’s it for this week! I plan to have some smaller quality-of-life changes and better updates coming soon…

Sooooo this devlog is mostly comprised of website changes. I have a conference in San Francisco and wanted to polish up the app before publishing. Here are the main changes I made:

13. Random chores and updating the website

I absolutely love shadcn-ui, but so do AI models. In past reviews by stardance raters, as well as family and friends, they pointed out the website looked AI-generated. Hence, I decreased the number of gradients and added basic animations to make the site feel more alive. This was a first for me to implement animations, but the Tailwind docs made it quite simple. I used a specific code-like font instead of the generic fonts I had and tidied up CSS classes. The biggest change was adding a blurred-geometric background for a nicer feel.

14. Redesigning the website completely

I felt it was better to update the website as a whole, so I changed up the user interface. When I tested the app with real users, they complained that they didn’t know how to actually use the app. I changed the hero design to have clearer text and added an SVG-animation component showing how the site actually works. The feature section stayed the same, but there was a highlights grid (inspo https://cider.sh/) to highlight the main features. The FAQ was shortened to my liking, and I pulled some utilities from Tailwind for animations.

In terms of images, I updated the images of the app featured on the website and cleaned up some of the dimensions related to that CSS. The old homepage still lives inside OldHome() but isn’t used anymore.

15. Tidying up the README and build workflows

I edited the README to remove redundant information and make it more developer-focused. I also updated some information regarding GPU builds. For the build workflow, I ensured it would only run if the commit ended in “rebuild”. This ensures unnecessary build minutes are not spent on small changes not requiring a full rebuild.

16. Wake lock API fallback

This app aims to support all devices, but older devices have…funny APIs to say the least. For WebRTC, I implemented the legacy encoded streams API. However, the device would still turn off for seemingly no reason while the video was playing. On older devices, the wake lock API is not supported. However, there’s a unique workaround for this: any video element playing on the webpage will keep the screen from dimming. Hence, I used the NoSleep library to create an off-screen 1-second video element that keeps looping, ensuring the display doesn’t go dark. Some very fun stuff with browsers.

That’s it for this devlog! Mostly minor changes, but the next one will have better SW encoding support among other changes. Stay tuned…

I met with some friends to discuss my app…and realized I gotta lock in with the features. Among the various topics we discussed like the difficult UI or lack of automatic updates, they really wanted remote control features. So, I set out to make the remote display something I can actually touch, click, and type into. And patching some other bugs along the way.

9. Remote input

This is the main headline feature. Streaming pixels one way was solved; now the client had to send input back up the wire and have the host replay it as if it came from real hardware.

Host side (Windows). A new input backend injects events through a synthetic pointer device, so mouse, touch, and pen all seem real as opposed to faked mouse moves. The synthetic device can execute touch or mouse movements from the tablet client. macOS and Linux backends are stubbed and will come later (trust me bro).

The wire. This side had to be optimized for dead-low latency like the rest of the project. Instead of passing strings or using some wrapper library, I made my own binary protocol (a dedicated protocol module) serialized over WebRTC data channels, not the media path. Client-side input.js captures keyboard, mouse, pointer (touch/pen), wheel, clipboard, and drag-drop and ships them across. The code basically catches every single frontend input event and forwards all those events (20+)! The routing is latency-first, same as everything else, with three different paths. fast-path events (pointer moves, wheel) take an unreliable channel, where a dropped packet is cheaper than a late one, while clicks, keys, and clipboard take the reliable channel where every event must land exactly once. It’s not that big of a deal if one small mouse delta doesn’t go through, but it definitely is a big deal if a click doesn’t go through.

10. Browser compatibility detections

Not every browser can do WebCodecs, secure contexts, or the input paths above. Rather than fail silently, I added a compatibility check on both the browser and client up front. It shows compatibility modals with per-feature lists to ensure the user knows what’s going on.

11. Some more small fixes

I added two quality-of-life additions: avatars (upload custom profile pictures) and zoom controls (ctrl/cmd +/-).

12. Encoders that don’t explode when funny stuff happens

Oh my god…one of the biggest issues by far. When I ran the NVIDIA encoder at high FPS, it didn’t crash. However, the Intel Quick Sync encoder would arbitrarily crash at high FPS and wouldn’t give a reason why. I audited a buncha different parts of my code and eventually found that FPS was the issue, not resolution or low-latency flags.

Hence, I added a binary search that probes the hardware for its true maximum supported fps and clamps to it before the encoder can choke. Essentially, I ask the encoder “can you do 1920x1080 at 240 fps?” –> “no” –> “what about 120 fps?” –> “yes” –> “what about 180 fps?”. This keeps repeating till the max FPS is found and streamed to the client.

Rounding out the release, I made a variety of different UI fixes like the modal fixes, backdrop overlap, a stray uninstall-modal backdrop, and cloud reconfig errors. More progress to come!

ScreenExtend Devlog: Building the Rust Engine Under a Wireless Second Monitor

TL;DR

ScreenExtend turns any device with a browser into a real, wireless extended monitor for a Windows host, no client app and no cables (it began so my brother could use his iPad for notes at debate tournaments). Scan a QR code and the host spins up a real virtual display, streamed over WebRTC with hardware H.264. The React/TS control UI existed before; everything else (Rust core, GPU pipeline, WebCodecs client, cloud relay) was built from scratch, always latency-first, quality-second.

1. The spine: Tauri + a typed Rust↔TS bridge

tauri-specta exports a fully-typed src/lib/bindings.ts for commands (TS→Rust) plus typed events (Rust→TS): devices, network, cloud status, logs. A code-generated bridge beats maintaining one by hand. Let the code write itself!

2. Virtual displays: faking a real monitor

Each client gets its own virtual display the OS treats as real hardware, a true second screen, not mirroring. On join the host creates one sized to the client, forces extend topology, then captures it once attached. On Windows a signed IDD driver installs at runtime (via nefconc + certutil), and it tears down cleanly on leave.

3. First light: WebRTC + WHEP + H.264

The media path is WebRTC (real-time, in every browser). Signaling is WHEP: the client POSTs an SDP offer, the host answers in-response, no signaling server. Codec is H.264, universal hardware decode on clients and encode (NVENC/QSV/AMF) on hosts. The first release streamed NVENC to a plain <video>; it worked but was laggy. It binds HTTP + HTTPS (self-signed via rcgen), since WebCodecs needs a secure context.

4. The pipeline: capture → encode → broadcast

Capture is Windows Graphics Capture (later a custom fork; DXGI Desktop Duplication fallback), with a repeater thread for idle keepalive/IDRs and transient-stall tolerance.

5. Going vendor-specific: the part that made it fast

Generic capture→CPU-copy→encode is slow everywhere; the win: keep the frame on the GPU from capture to bitstream. Per vendor:

• NVENC (NVIDIA): no SDK linked (API dynamically loaded); zero-copy writes capture into a shared D3D11 texture behind a keyed mutex, else a CPU bridge.
• Intel QSV/oneVPL: same-adapter capture+encode fuses the downscale into the VPP pass (BGRA→NV12 + resize in one shot).
• macOS (VideoToolbox): ScreenCaptureKit (12.3+, CGDisplayStream fallback) via objc2 runtime interop only, so one binary loads on every macOS version, same IOSurface zero-copy.

6. The client: WebCodecs over <video>

Fast path: an RTCRtpScriptTransform feeds a worker VideoDecoder (optimizeForLatency: true) onto an OffscreenCanvas, latency hints zeroed. A <video> fallback covers missing WebCodecs or stalls. It also feels like a monitor: fullscreen + input/wake locks, /leave beacon on exit.

7. Adaptive bitrate, sessions & security

A BWE driver reads getStats every 250ms, pushing a smoothed bitrate into the encoder. Changing settings triggers in-place renegotiation (a reconfig/kick epoch the client polls) with no display teardown. Each host is gated by a session ID + 6-digit OTP (OtpLimiter lockout); every device is isolated, over HTTPS + DTLS/SRTP.

8. Networking + status (v0.2.3)

Same-network is direct, no servers. Offline mode stands up an ad-hoc hosted network, no infrastructure needed. Cross-network adds a cloud relay (WebSocket to session.screenextend.app) tunneling signaling while media stays P2P; a self-hosted TURN server covers hard NATs. A logbus feeds an in-app terminal. Working today: Windows/macOS hosts, per-vendor GPU encode, per-device displays, adaptive bitrate, OTP sessions, offline mode, and cloud relay.