Dev Log #3 — SmolLM3-3B Results

SmolLM3-3B was the first real test of ThinTensor as an inference runtime.

The baseline was Hugging Face BF16 on my RTX 5050 laptop GPU:

HF BF16: ~17.8 tok/s

ThinTensor BF16 reached:

ThinTensor BF16: ~32–33 tok/s

with cosine around 0.999, meaning the logits stayed extremely close to the original SafeTensors/HF path. That matters because a storage/runtime format should not make the model noticeably worse.

Then came the quality FP8 profile:

ThinTensor quality FP8: ~46–48 tok/s

This was not “quantize everything.” The profile was selective:

gate/up projections: FP8
middle down projections: FP8
attention path: BF16
O-proj: BF16
LM head: BF16
KV cache: BF16

ThinTensor can do this because the .thin manifest and execution tape know tensor roles: Q, K, V, O, gate, up, down, norm, embedding, LM head, etc.

The runtime can then lower precision only where the model tolerates it, while keeping sensitive paths in BF16.

The kernel side reflects this. Attention uses single-token GQA and maps each query head to the right KV head:

head = tl.program_id(0)
kv_head = head // (heads // kv_heads)

scores = tl.sum(key * q[None, :], axis=1) * scale

The FP8 MLP path uses scaled matvecs:

w = tl.load(weight + offs_m[:, None] * stride_wm + offs_n[None, :])
xv = tl.load(x + offs_n).to(tl.float32)
scale = tl.load(scales + offs_m).to(tl.float32)

acc = tl.sum(w * xv[None, :], axis=1) * scale

In one full-vocab centered-logit comparison:

GGUF Q8_0 min cosine:              0.952666
ThinTensor BF16 min cosine:        0.999596
ThinTensor quality FP8 min cosine: 0.995366

So ThinTensor quality FP8 stayed much closer to BF16 than the measured Q8 path, while staying in the same speed class.

Final speed table:

HF BF16:                 ~17.8 tok/s
ThinTensor BF16:         ~32–33 tok/s
ThinTensor quality FP8:  ~46–48 tok/s
Ollama / GGUF Q8_0:      ~44.1 tok/s

Headline:

ThinTensor BF16 was ~1.8x faster than HF BF16.
ThinTensor quality FP8 was ~2.6x faster than HF BF16.
ThinTensor quality FP8 was faster than the measured Q8 baseline while preserving much better logit fidelity.

This is where ThinTensor stopped being just an archive experiment and started looking like a real inference runtime :D