Devlog #2

Last time I said the plan is to store meaning, not the waveform, the way face recognition stores an embedding instead of the photo. That left a question I skipped over: how do you actually train a compressor to do that?

The naive version is to run the EEG through an encoder, get an embedding, and store just that. But it has two problems. First, I’d be throwing the raw signal away for good, and a brain recording is expensive and reusable, you might want to ask it a different question later. Second, and bigger: what target do I train the embedding toward? An autoencoder left to itself just learns whatever compresses easiest, which lands me right back at fidelity.

The insight I landed on: I don’t have to invent the meaning space. It already exists. Image and text models like CLIP have already learned a space where a picture of an accordion and the word “accordion” sit in the same spot. So instead of hoping my compressor discovers meaning on its own, I borrow a model that already has it and use it as a judge.

Here’s the shape. The codec compresses and rebuilds the EEG like a normal codec. Then I take the rebuilt signal, push it through a frozen model that maps it into that shared image-text space, and score how close it lands to the embedding of the image the person actually saw. If the reconstruction drifts away from the right meaning, that score punishes it. The judge never changes, it only grades.

The important part is that the gradient has to flow through the frozen judge back into the compressor. The judge is frozen but not detached: it’s a fixed grader whose opinion shapes what the codec learns to keep. That one detail is easy to get wrong, and it’s the whole trick.

So the loss becomes three things added together: spend few bits, rebuild the waveform roughly, and stay decodable in the judge’s space. That third term is the entire idea. It’s what makes the codec hold onto the accordion instead of the loudest wiggle in the signal.

And because the judge lives in a text-image space, search comes for free. Type a word, it becomes a point in the same space, and I rank the stored recordings by how close they land. I never trained on text.