Initial benchmarks based on some testing on a 3B model

Why response similarity?

Because we are making a storage format not “just” a runtime, so we need to ensure the model we are optimizing using thintensor runtime it HAS to be replying with the same responses like the original model!

So with casual Kv implemented these are the benchmarks i gathered, if you wanna learn what quality fp8 is read my last devlog :D

PS- the vram usage on our ThinTensor quality FP8 is just 500 mb more than the q8 even though there is a HUGE difference in the response similarity