Dev Log #4 — 91–96 tok/s with Selective MXFP4

This was the first ThinTensor run that felt genuinely fast.

On SmolLM3-3B, the selective MXFP4 gate/up profile reached around:

~91–96 tok/s

and on the strong tested record it still preserved:

cosine_similarity: 0.9994624257087708
top1_same: true
top5_overlap: 0.8
generated_token_match_rate: 0.875
KV retention: full BF16

That cosine is extremely close to the HF BF16 reference:

1.0 - 0.9994624257087708 = 0.0005375742912292

So the run was only about 0.054% away from perfect cosine on that tested checkpoint.

Why this was not just normal quantization

The profile was selective:

MXFP4 gate/up layers: 0:24
MXFP4 row postscale: enabled
LM head: FP8 guarded path
KV cache: full BF16
attention path: causal KV
QKV/O/down: not blindly pushed into MXFP4

A full MXFP4 sweep over QKV, O, down, and gate/up was fast, but it destroyed the logits. That proved the obvious thing: fewer bytes alone is not enough.

ThinTensor’s win comes from being role-aware.

The runtime knows what each tensor does:

Q / K / V
O projection
gate projection
up projection
down projection
norm
embedding
LM head
KV cache

That means the runtime can apply aggressive precision only to the parts that tolerate it.

For this result, the safe target was the MLP gate/up path.

Why gate/up was the right target

Single-token decoding is mostly a weight-read problem.

Tthe goal became:

reduce gate/up bytes
keep attention stable
keep KV exact
keep LM head guarded
preserve token ranking

That is exactly what the selective MXFP4 profile does.

It uses MXFP4 on the gate/up projections for layers 0:24, but it does not blindly lower every sensitive path.

Where Blackwell helps

The test GPU is an RTX 5050 laptop GPU, which is Blackwell.

That matters because Blackwell has native low-precision paths for FP4/MXFP4-style execution. ThinTensor uses this through Triton kernels that treat MXFP4 as an execution format, not just a smaller storage format.

The idea is:

store/read fewer bytes
use block scales
multiply against BF16 activations
accumulate safely
then apply row postscale

The goal is to make each token cheaper to decode while keeping the output distribution close to HF BF16.

The code side

The relevant code is built around these ideas:

1. Honest quantization descriptors

The QuantizationDescriptor describes the source format:

method
storage_dtype
bits
group_size
scale_dtype
zero_point
modules_to_not_convert
source_config

For example:

INT4 != FP4
Q8 != FP8
MXFP4 != generic Q4

The precision_ladder() function makes that explicit. It lists safe choices, opt-in choices, and lossy choices.

So instead of silently lowering precision, the runtime can say:

preserve native format
fallback to BF16
or require explicit validation before requantization

2. Execution planning

plan_quantization() decides what should happen at runtime.

It can choose:

preserve source storage
use native kernel
dequantize to BF16
or reject unsupported requantization

The important flag is this:

exact_storage_preserved

If this is true, the runtime is using the source encoding directly.

Whats next

I would have to implement all mainstream model archs manually because all are diff, and like right now you can see, there are a lot of models that are heavily un-optimized

Also im messaging stardance team on slack these devlogs too smol for me