ScreenExtend Devlog: Building the Rust Engine Under a Wireless Second Monitor

TL;DR

ScreenExtend turns any device with a browser into a real, wireless extended monitor for a Windows host, no client app and no cables (it began so my brother could use his iPad for notes at debate tournaments). Scan a QR code and the host spins up a real virtual display, streamed over WebRTC with hardware H.264. The React/TS control UI existed before; everything else (Rust core, GPU pipeline, WebCodecs client, cloud relay) was built from scratch, always latency-first, quality-second.

1. The spine: Tauri + a typed Rust↔TS bridge

tauri-specta exports a fully-typed src/lib/bindings.ts for commands (TS→Rust) plus typed events (Rust→TS): devices, network, cloud status, logs. A code-generated bridge beats maintaining one by hand. Let the code write itself!

2. Virtual displays: faking a real monitor

Each client gets its own virtual display the OS treats as real hardware, a true second screen, not mirroring. On join the host creates one sized to the client, forces extend topology, then captures it once attached. On Windows a signed IDD driver installs at runtime (via nefconc + certutil), and it tears down cleanly on leave.

3. First light: WebRTC + WHEP + H.264

The media path is WebRTC (real-time, in every browser). Signaling is WHEP: the client POSTs an SDP offer, the host answers in-response, no signaling server. Codec is H.264, universal hardware decode on clients and encode (NVENC/QSV/AMF) on hosts. The first release streamed NVENC to a plain <video>; it worked but was laggy. It binds HTTP + HTTPS (self-signed via rcgen), since WebCodecs needs a secure context.

4. The pipeline: capture → encode → broadcast

Capture is Windows Graphics Capture (later a custom fork; DXGI Desktop Duplication fallback), with a repeater thread for idle keepalive/IDRs and transient-stall tolerance.

5. Going vendor-specific: the part that made it fast

Generic capture→CPU-copy→encode is slow everywhere; the win: keep the frame on the GPU from capture to bitstream. Per vendor:

• NVENC (NVIDIA): no SDK linked (API dynamically loaded); zero-copy writes capture into a shared D3D11 texture behind a keyed mutex, else a CPU bridge.
• Intel QSV/oneVPL: same-adapter capture+encode fuses the downscale into the VPP pass (BGRA→NV12 + resize in one shot).
• macOS (VideoToolbox): ScreenCaptureKit (12.3+, CGDisplayStream fallback) via objc2 runtime interop only, so one binary loads on every macOS version, same IOSurface zero-copy.

6. The client: WebCodecs over <video>

Fast path: an RTCRtpScriptTransform feeds a worker VideoDecoder (optimizeForLatency: true) onto an OffscreenCanvas, latency hints zeroed. A <video> fallback covers missing WebCodecs or stalls. It also feels like a monitor: fullscreen + input/wake locks, /leave beacon on exit.

7. Adaptive bitrate, sessions & security

A BWE driver reads getStats every 250ms, pushing a smoothed bitrate into the encoder. Changing settings triggers in-place renegotiation (a reconfig/kick epoch the client polls) with no display teardown. Each host is gated by a session ID + 6-digit OTP (OtpLimiter lockout); every device is isolated, over HTTPS + DTLS/SRTP.

8. Networking + status (v0.2.3)

Same-network is direct, no servers. Offline mode stands up an ad-hoc hosted network, no infrastructure needed. Cross-network adds a cloud relay (WebSocket to session.screenextend.app) tunneling signaling while media stays P2P; a self-hosted TURN server covers hard NATs. A logbus feeds an in-app terminal. Working today: Windows/macOS hosts, per-vendor GPU encode, per-device displays, adaptive bitrate, OTP sessions, offline mode, and cloud relay.