Ambient Backlight for monitor

I improved the GPU readback so the CPU no longer has to wait for the GPU every frame (Gemini noticed that this happend and with its help I fixxed it). Before the resul buffer was copied and read immediately which could cause the CPU to block while the GPU was still processing shader.

This is difficult because getting the CPU and GPU sync right is tough. As the CPU should only read a buffer when the GPU finished writing on it.
I used ID3D11Query events for it:

if (r.pending &&
context->GetData(r.query.Get(), nullptr, 0, 0) == S_OK)
{
context->Map(r.buffer.Get(), 0, D3D11_MAP_READ, 0, &m);
serial.send((const UINT*)m.pData);
context->Unmap(r.buffer.Get(), 0);
r.pending = false;
}

Fixed a few issues in the shader. First of all the border mapping wasnt complete and I forgot the top and left side of the LEDs.. Also removed Typos. I also added sRGB and improved the saturation handling as before they didnt match accurate. I fixed it by the colors to linear space before processing them.

Before:

float luma = dot(rgb, float3(0.2126, 0.7152, 0.0722));
rgb = saturate(lerp(luma.xxx, rgb, saturationBoost) * brightness);

After:

sum += ToLinear(desktop.Load(int3(pixel, 0)).rgb);

float3 rgb = ToSrgb(sum / (samplesAcross * samplesDeep));
rgb = saturate(BoostSaturation(rgb) * brightness);

Using an HLSL compute shader and [numthreads(90, 1, 1)] the GPU spawns 90 parralel threads which are calculating the average color for every LED simaultaneously.

As only picking one pixel at each led would be pretty inaccurate, we are taking zones and use the average color of ever zone.

(For now the average of every zone is picked, if I am testing it in real
life, we might change it - maybe the outer pixels are doubled for a more accurate color that looks better at the end).

I had an issue where some LEDs were using the wrong part of the screen because of incorrect index calculation. I fixed it by correcting the LED and screen coordinates:

(in the picture)

Before doing the PC side and video capture code, I made the the code for the physical ahrdware (the esp32). I set up a Arduino sketch using FastLED library to manage the output signal on GPIO (in my case DATA_PIN 2), I configured the setup for the 90 LEDS I have on the RGB Strip (ws2812b usually have 60 LEDS per 1m and i need 1,5m). I set the communication speed to 921.600 baud.

void setup() {    FastLED.addLeds<WS2812B, DATA_PIN, GRB>(leds, NUM_LEDS);    FastLED.setBrightness(BRIGHTNESS);    FastLED.clear(true);    Serial.begin(921600);}

To avoid lagging, I built a simple step by step parser that handles incoming data byte by byte:

1. Ready how many Leds are sent
2. It Checks if the data is valid and mathces the 90 LEDS and if anything is wrong it resets and waits for the next frame.
3. Once all 270 bytes (90 lights * 3 colors) arrive, it calls FastLED.show() to light up the strip

I started working on the communication between the PC and the LED hardware with serial. Serial is the channel used to send binary color data directly to the esp32.
To make sure the lights dont stay lit when the program closes the destructor sends a black frame on exit to turn of all lights. I didnt notice that it would be a problem but when troubleshooting for another proble, gemini noticed it:

~Serial() { black(); if (handle_ != INVALID_HANDLE_VALUE) CloseHandle(handle_); }

I am using DXGI Desktop duplication that can capture the frames directly from the gpu. This is good so the frames dont need to be loaded again for the LEDs.

I started with creating the circuit in KiCad. I noticed that there was not the right ESP32 for me so i looked online and found this:
https://github.com/davidkleymann/doit-esp32-devkit-kicad

I build everything in KiCad just to see that the libraries are probably outdated (or i just did something wrong but i dont believe that)
After about 20 of trouble shooting I got it working but I noticed that KiCad is not the best place for what im trying to do so I gave up. I searched for a new programm and found https://wokwi.com.

It worked well. Though, they did not have splicing connectors / wago connectors so i made it with a Breadboard in this circuit instead. In the real version I will do it instead with splicing connectors.
Also the LED strip name for the negative power supply pin was VSS, i thought it should be GND and learned the difference then.