@EOBD

ohhhhh MY

ooh My

I got CDNA4

And GLM5.3

it’s a party NOW!!

I’ma first find the best way to quantize the experts
than we go swimming in Aiter to find A8W4 for the experts and A8WA for the rest

we then benchmark plain torch – probably 20% work for 80% performance
we then squeeze the residuals into a fused kernel + proper grouped gemm for the MLA + fp4 indexer keys

and then, finally, see if it all comes under one launch, and does a pretty spin to become a 100% resident, HBM synced pipeline system. it’s no longer a kernel, but an async token horse

I did fail my tests but worth it I tell ya!

Huge Pain with tile schedule for the MLP. huge.
still haven’t fully figured it out, but getting there

Don’t leave XNACK on 🤦🏽

It’s been like a whole day trying to understand why I have these idles (not a stall)
Hate myself so much.

And mannnn – AMD – why nooooo documentation whatsoever as to how yor MMU works ?? 😭

not sure if I’d feel better, learning that they are willing to share this – but you’d just need to sign off your grandma as collateral on an NDA

Or maybe show somewhere that there was a fault 😭😭😭 (could also be me not knowing where to look…)

….but…. while breaking my head, I also improved rocprof compute viewer 🎉
and it’s going GPL until AMD gives me access to CDNA4 and 5 hardware😇 Because they are so so nice and willing to support a budding kernel developer ♥️
or I’ll stop being lazy and write a PR
which ever comes first I guess

Going to do the MoE next (HipKittens sync first)

And so before I commit to a very complex design that would piggyback low-token-experts on ones that can fill the 16 rows for the matrix cores – though I should what the distrebution actually looks like…

): complex design will be needed ):

Expected but still a bummer.

My sample space is tiny because GPUs are expensive as hell.

But I imagine it’s a good proxy because the curve looks right. maybe a bit more concentrated across a large chat focused on coding.

Well.
That’s been a huge pain.

But I’s got a native unix shell
Should be amazing to not rely on this slugish p9

windows is like sweet poison
so nice
but so sloowwwww

bitlocker is scary guys

ubuntu is the best – linux without the added pain of knowing how linux works (:

last 6 spills super nasty. LLVM be complex
should also hoist the loads before the stores (ugly stalls are swaits).
but look at that pretty no-sigpgmend-barrier work

I don’t hate LLVM anymore!!

So easy to just hook up to its headers and do what the hell I want with it (mostly in terms of what opt passes to run on the device code because I’m scared of that fat binary linking 😭)

LICM is being stupid? Kick that sucker off the top the stairs!!

possibly could have also done this with opt but C++ is way cooler and going to pay off once I really dig deep into this.

Ima get my 0 spills even if it’s going to cost me all my finals. probably already did 🥲

phhh that poor-people normal kernel… Hardware scheduled… Meh. No match for my persistent kernel!

not actually sure why I beat it, shouldn’t have. Stores are not acked on at the end of the kernel.
But we first smile, then run them back to back and earn that smile! Look at me utilizing smile debt like finance bro.

Maybe its my very cool scheduling algo. but I doubt it.

I know how I’ll do it!
We will unfraternally have to have kernel boundaries
but really just because the L1i cache is too small (and GPU architecture is too dumb)
But it’s fine for now.

Going to have the following kernels:
1:1 Residual + Norm
8:1 Residual + Norm (for the MoE summing)
1:1 Residual + Norm + MPT norm
MoE
Full DSA Fused attention (with or without indexer)
Head + Sampling +Embedding retrieval + normed MTP embedding
EH_Proj + MLP

And then!
For the HBM sync we need between so many things (mostly the attention kernel)
I’m going to extend HipKittens with a flagging mechanism that latches onto the stores, and add many more load methods that depend on it (like a stream of available blocks. the coolest thing is that I can have the flagging work at a certain granularity, and the load can infer if it’s OK based on a different one!

Don’t do kernels kids.

(90h) I don’t like LLVM! fun! Have you ever written manual ASM loops because it hoists like there’s no tomorrow?
and can’t handle a back edge??

And since it’s impossible to get CDNA4 hardware, will be moving forward with MI300s.

All my ❤️ to RunPod and HotAisle (and AMD dev cloud)
Still a bit annoying because I much rather pay 5x and get access to CDNA4 then free credits. But also nice ! THANKS !

And to whoever is hogging all the capacity from from cloudrift – pig (:

EDIT: llvm is very nice
I love anyone who has ever worked on LLVM
it was the AMD backend hoisting my licm
figured it out though

And ASM loops are a really bad idea