Going to do the MoE next (HipKittens sync first)

And so before I commit to a very complex design that would piggyback low-token-experts on ones that can fill the 16 rows for the matrix cores – though I should what the distrebution actually looks like…

): complex design will be needed ):

Expected but still a bummer.

My sample space is tiny because GPUs are expensive as hell.

But I imagine it’s a good proxy because the curve looks right. maybe a bit more concentrated across a large chat focused on coding.