I know how I’ll do it!
We will unfraternally have to have kernel boundaries
but really just because the L1i cache is too small (and GPU architecture is too dumb)
But it’s fine for now.

Going to have the following kernels:
1:1 Residual + Norm
8:1 Residual + Norm (for the MoE summing)
1:1 Residual + Norm + MPT norm
MoE
Full DSA Fused attention (with or without indexer)
Head + Sampling +Embedding retrieval + normed MTP embedding
EH_Proj + MLP

And then!
For the HBM sync we need between so many things (mostly the attention kernel)
I’m going to extend HipKittens with a flagging mechanism that latches onto the stores, and add many more load methods that depend on it (like a stream of available blocks. the coolest thing is that I can have the flagging work at a certain granularity, and the load can infer if it’s OK based on a different one!

Don’t do kernels kids.