Final benchmarks

Smaller models have lower cosine because there are like really small like 1B but, some archs are still not fully optimized to run at a good bit faster speed, ik, but i would want to do a ship before working on getting the speed up on those

Finished benchmarking a few more models

Hitting really really good tk/s when compared to hf that even fails to load the models on my 8gb vram, so its a HUGE improvement for gpu poor ppl like me 😭

Reason a lot less improvement for many models is that, each model has a different structure and i havent been able to make a generalized algorithm that would work for all, so for now, we are working with just a simple implementation

Benchmark Methodology and Metrics Calculation

To ensure accurate, robust, and reproducible results, the benchmarking and correctness scoring flow is executed as follows:

1. Benchmark Execution Workflow

Benchmarks are run in isolated, dedicated Python processes for each engine (ThinTensor vs. Hugging Face Transformers) to prevent memory leakage or process interference:

• ThinTensor Baseline:
  • The model is loaded via the thin_runtime.py engine.
  • Weight Fit Planning: The runtime runs _automatic_fit_plan to estimate weight size. If the model’s weights do not fit within the available GPU VRAM (accounting for system/display manager overhead), it dynamically configures streaming weight residency (--weight-residency stream) and streams layers/experts block-by-block from CPU/RAM to the GPU. If the weights fit, it uses fully resident weights (--weight-residency all).
  • Timing: Prefills the prompt (default “Hello”), decodes a number of warmup tokens (default 10) to initialize caches and compile/warm up Triton kernels, and then times the next decode steps (default 200) using wall-clock time (time.perf_counter()).
• Transformers (Hugging Face) Baseline:
  • The model is loaded via a dedicated script bench_hf_transformers_decode.py in BF16 precision.
  • RAM/CPU Offloading: For larger models (e.g. Gemma-4-E2B, OLMoE), loading the model fully on memory-constrained GPUs (like the RTX 5050 Laptop GPU with 7.57 GiB VRAM) causes CUDA OOM. To prevent OOM, RAM offloading is enabled via device_map="auto". Weights that exceed VRAM capacity are kept in CPU system RAM and loaded/swapped as needed.
  • Timing: Like ThinTensor, it pre-runs a prompt prefill, decodes warmup tokens (default 10), and then times steps (default 200) decode steps. The decode loop avoids GPU-to-CPU synchronization (e.g. calling .item() inside the timed loop) to measure pure GPU decoding speed.

2. Metrics & Scores Calculation

• Thin/HF Peak GiB (Peak GPU Memory): Tracks peak allocated VRAM in GiB using torch.cuda.max_memory_allocated(). This measures active tensors allocated by PyTorch’s allocator during the decode phase (excluding unallocated cache reserve or general display server memory).
• Min Cosine (Cosine Similarity): In compare_hf_thin_logits.py, the logits of the final linear projection layer are collected for both models. Cosine similarity is calculated for the target token index at each step. The minimum cosine similarity found across all decode steps (e.g., step 1 and step 10) and prefill lengths (e.g., 1 and 128) is reported.
• Top-1 Match: A boolean check indicating whether the token ID with the highest probability (highest logit value) is identical between ThinTensor and HF at all decode steps.
• Top-5 Set: A boolean check indicating whether the set of top-5 highest-probability token IDs is identical between the two models at all steps (ignoring their internal sorted order).
• Top-5 Order: A boolean check indicating whether the exact sorted order of the top-5 token IDs matches between ThinTensor and HF at all steps.
• KV Retention: Specifies the KV cache precision format (e.g. full BF16).

Dev Log #4 — 91–96 tok/s with Selective MXFP4

This was the first ThinTensor run that felt genuinely fast.

On SmolLM3-3B, the selective MXFP4 gate/up profile reached around:

~91–96 tok/s

and on the strong tested record it still preserved:

cosine_similarity: 0.9994624257087708
top1_same: true
top5_overlap: 0.8
generated_token_match_rate: 0.875
KV retention: full BF16

That cosine is extremely close to the HF BF16 reference:

1.0 - 0.9994624257087708 = 0.0005375742912292

So the run was only about 0.054% away from perfect cosine on that tested checkpoint.

Why this was not just normal quantization

The profile was selective:

MXFP4 gate/up layers: 0:24
MXFP4 row postscale: enabled
LM head: FP8 guarded path
KV cache: full BF16
attention path: causal KV
QKV/O/down: not blindly pushed into MXFP4

A full MXFP4 sweep over QKV, O, down, and gate/up was fast, but it destroyed the logits. That proved the obvious thing: fewer bytes alone is not enough.

ThinTensor’s win comes from being role-aware.

The runtime knows what each tensor does:

Q / K / V
O projection
gate projection
up projection
down projection
norm
embedding
LM head
KV cache

That means the runtime can apply aggressive precision only to the parts that tolerate it.

For this result, the safe target was the MLP gate/up path.

Why gate/up was the right target

Single-token decoding is mostly a weight-read problem.

Tthe goal became:

reduce gate/up bytes
keep attention stable
keep KV exact
keep LM head guarded
preserve token ranking

That is exactly what the selective MXFP4 profile does.

It uses MXFP4 on the gate/up projections for layers 0:24, but it does not blindly lower every sensitive path.

Where Blackwell helps

The test GPU is an RTX 5050 laptop GPU, which is Blackwell.

That matters because Blackwell has native low-precision paths for FP4/MXFP4-style execution. ThinTensor uses this through Triton kernels that treat MXFP4 as an execution format, not just a smaller storage format.

The idea is:

store/read fewer bytes
use block scales
multiply against BF16 activations
accumulate safely
then apply row postscale

The goal is to make each token cheaper to decode while keeping the output distribution close to HF BF16.

The code side

The relevant code is built around these ideas:

1. Honest quantization descriptors

The QuantizationDescriptor describes the source format:

method
storage_dtype
bits
group_size
scale_dtype
zero_point
modules_to_not_convert
source_config

For example:

INT4 != FP4
Q8 != FP8
MXFP4 != generic Q4

The precision_ladder() function makes that explicit. It lists safe choices, opt-in choices, and lossy choices.

So instead of silently lowering precision, the runtime can say:

preserve native format
fallback to BF16
or require explicit validation before requantization

2. Execution planning

plan_quantization() decides what should happen at runtime.

It can choose:

preserve source storage
use native kernel
dequantize to BF16
or reject unsupported requantization

The important flag is this:

exact_storage_preserved

If this is true, the runtime is using the source encoding directly.

Whats next

I would have to implement all mainstream model archs manually because all are diff, and like right now you can see, there are a lot of models that are heavily un-optimized

Also im messaging stardance team on slack these devlogs too smol for me

Just hit 91tk/s with “cosine_similarity”: 0.9994624257087708

its actually crazy its like 5x faster the hf full devlog comming soon!

ITS LIKE 0.054% away from perfect!!!

The selective MXFP4 gate/up profile reached ~96 tok/s while preserving top-1 on the tested SmolLM3-3B through my thintensor runtimeeeee

Full DevLogs gonna be crazy

Initial benchmarks based on some testing on a 3B model

Why response similarity?

Because we are making a storage format not “just” a runtime, so we need to ensure the model we are optimizing using thintensor runtime it HAS to be replying with the same responses like the original model!

So with casual Kv implemented these are the benchmarks i gathered, if you wanna learn what quality fp8 is read my last devlog :D

PS- the vram usage on our ThinTensor quality FP8 is just 500 mb more than the q8 even though there is a HUGE difference in the response similarity

Dev Log #3 — SmolLM3-3B Results

SmolLM3-3B was the first real test of ThinTensor as an inference runtime.

The baseline was Hugging Face BF16 on my RTX 5050 laptop GPU:

HF BF16: ~17.8 tok/s

ThinTensor BF16 reached:

ThinTensor BF16: ~32–33 tok/s

with cosine around 0.999, meaning the logits stayed extremely close to the original SafeTensors/HF path. That matters because a storage/runtime format should not make the model noticeably worse.

Then came the quality FP8 profile:

ThinTensor quality FP8: ~46–48 tok/s

This was not “quantize everything.” The profile was selective:

gate/up projections: FP8
middle down projections: FP8
attention path: BF16
O-proj: BF16
LM head: BF16
KV cache: BF16

ThinTensor can do this because the .thin manifest and execution tape know tensor roles: Q, K, V, O, gate, up, down, norm, embedding, LM head, etc.

The runtime can then lower precision only where the model tolerates it, while keeping sensitive paths in BF16.

The kernel side reflects this. Attention uses single-token GQA and maps each query head to the right KV head:

head = tl.program_id(0)
kv_head = head // (heads // kv_heads)

scores = tl.sum(key * q[None, :], axis=1) * scale

The FP8 MLP path uses scaled matvecs:

w = tl.load(weight + offs_m[:, None] * stride_wm + offs_n[None, :])
xv = tl.load(x + offs_n).to(tl.float32)
scale = tl.load(scales + offs_m).to(tl.float32)

acc = tl.sum(w * xv[None, :], axis=1) * scale

In one full-vocab centered-logit comparison:

GGUF Q8_0 min cosine:              0.952666
ThinTensor BF16 min cosine:        0.999596
ThinTensor quality FP8 min cosine: 0.995366

So ThinTensor quality FP8 stayed much closer to BF16 than the measured Q8 path, while staying in the same speed class.

Final speed table:

HF BF16:                 ~17.8 tok/s
ThinTensor BF16:         ~32–33 tok/s
ThinTensor quality FP8:  ~46–48 tok/s
Ollama / GGUF Q8_0:      ~44.1 tok/s

Headline:

ThinTensor BF16 was ~1.8x faster than HF BF16.
ThinTensor quality FP8 was ~2.6x faster than HF BF16.
ThinTensor quality FP8 was faster than the measured Q8 baseline while preserving much better logit fidelity.

This is where ThinTensor stopped being just an archive experiment and started looking like a real inference runtime :D

Js got to know that devlogs longer than 10hrs are stripped to 10 😭, i cant have more than 4k char so bear with me theres a LOT to tell that im not able to fit, but ill try my best anyways

Dev Log #2 — Converting SafeTensors Into .thin

(had to cut so much explaination bcs i HATE THE CHAR LIMIT, id add them in another devlog)

The main conversion flow is:

pub fn convert_hf(options: ConvertHfOptions) -> Result<ConvertHfResult> {
    let config: Value = serde_json::from_reader(
        File::open(options.hf_dir.join("config.json"))?
    )?;

    let safetensor_paths = collect_safetensors(&options.hf_dir)?;
    let tensors = collect_tensors(&safetensor_paths)?;

    let layers = required_u32(&config, "num_hidden_layers")?;
    let model = build_model_spec(&config, layers, &tensors, &options, &mut warnings)?;
    let pages = build_pages(&tensors, &config)?;
    let execution_tape = build_execution_tape(layers, &tensors, &mut warnings)?;

    let manifest = Manifest {
        format: FORMAT_NAME.to_string(),
        version: FORMAT_VERSION,
        model,
        execution_tape,
        pages,
        memory_plan: build_memory_plan(&config, &tensors),
    };

    write_archive_pages(&options.out_path, &manifest, &archive_pages)?;
    let archive = Archive::open(&options.out_path)?;
    verify_archive(&archive)?;

    Ok(ConvertHfResult { archive, warnings })
}

The converter memory-maps each SafeTensors file read-only:

let mmap = unsafe { Mmap::map(&file)? };
let safetensors = SafeTensors::deserialize(&mmap)?;
let base = mmap.as_ptr() as usize;

Then every tensor becomes a page candidate. The converter records where the tensor bytes live inside the mapped file:

let data = tensor.data();
let offset = (data.as_ptr() as usize).checked_sub(base)?;

and stores:

struct TensorPage {
    id: String,
    path: PathBuf,
    offset: u64,
    size: u64,
    checksum: [u8; 32],
    dtype: String,
    shape: Vec<u64>,
}

This avoids rebuilding every tensor in memory. The archive can later copy the original byte range directly:

ArchivePageSource::FileRange {
    path: tensor.path.clone(),
    offset: tensor.offset,
}

After collecting raw tensor data, the converter builds the model spec. It extracts or infers things like hidden size, layer count, attention heads, KV heads, head dimension, RoPE settings, vocab size, activation type, quantization config, and required runtime operators.

One important field is attention type:

let attention_kind = if kv_heads == heads {
    "mha"
} else if kv_heads == 1 {
    "mqa"
} else {
    "gqa"
};

This matters because many modern decoder models use grouped-query attention. If the runtime only sees shapes but does not understand Q heads vs KV heads, causal attention can be wrong while still looking structurally valid.

The converter also records required operators:

let mut required_operators = vec![
    "embedding".to_string(),
    "rms_norm".to_string(),
    "qkv_projection".to_string(),
    "rope".to_string(),
    format!("{attention_kind}_attention"),
    "o_projection".to_string(),
    "lm_head".to_string(),
];

For dense MLPs it adds:

gated_activation
down_projection

For MoE models it records router and expert stages instead.

the rest would be explained in another devlog, i hate the char limit

(had to re-write like 10 times btw)

Dev Log #1 — Why ThinTensor Exists

SafeTensors is already a very good format for what it was designed to do: store tensors safely. It keeps tensor metadata simple. For normal model distribution, that is enough.

The problem starts when the tensor file becomes the runtime boundary.

That is enough if another framework already knows the model class and execution graph. But for an inference runtime, especially one focused on single-token decode, those fields are not enough. The runtime needs to know what each tensor means.

The .thin Format

The first .thin archive is intentionally simple:

.thin = fixed header + JSON manifest + page table + raw page data

The Rust implementation defines the archive identity and header size directly:

pub const MAGIC: &[u8; 8] = b"THINv0\0\0";
pub const HEADER_LEN: u32 = 88;

The header stores the physical structure of the archive:

pub struct Header {
    pub header_len: u32,
    pub version: u32,
    pub manifest_off: u64,
    pub manifest_len: u64,
    pub page_table_off: u64,
    pub page_count: u64,
    pub data_off: u64,
    pub archive_hash: [u8; 32],
}

This lets the loader open the file without guessing. It reads the header, jumps to the manifest, then reads the page table, then maps the raw page data.

The manifest is JSON because the early format needed to stay inspectable. It contains the model-level information needed to reconstruct decode:

architecture
hidden size
number of layers
attention heads
KV heads
head dimension
vocabulary size
RoPE settings
tensor names
tensor shapes
tensor dtypes
tensor roles
quantization metadata
runtime/backend hints
validation metadata

The page table connects the manifest to real bytes inside the archive:

pub struct PageTableRecord {
    pub page_id: String,
    pub offset: u64,
    pub stored_size: u64,
    pub raw_size: u64,
    pub flags: u32,
    pub checksum: [u8; 32],
}

A page record gives the runtime a verified byte range, not just a tensor name. This matters because pages can later be scheduled, streamed, cached, kept on CPU, moved to GPU, or assigned different execution policies.

Packing and Verification

The packer does not blindly write files into an archive. It validates the manifest first:

let report = validate_manifest(&manifest);
if !report.is_ok() {
    bail!("manifest invalid:\n{}", report.errors.join("\n"));
}

Then every page declared in the manifest must exist and match its declared size:

if bytes.len() as u64 != page.size {
    bail!("page declared size does not match file size");
}

It also verifies the page checksum:

let checksum = blake3::hash(&bytes);
if checksum.as_bytes().as_slice() != expected.as_slice() {
    bail!("page checksum does not match manifest");
}

Only after validation does the writer emit the archive:

write_header(&mut file, &header)?;
file.write_all(&manifest_bytes)?;
for record in &records {
    write_page_table_record(&mut file, record)?;
}
for page in pages {
    write_page_source(&mut file, page)?;
}

So the final file is structured as:

[ header ]
[ manifest JSON ]
[ page table records ]
[ raw page blobs ]