Hitting 100hrs and final ship

Didnt get time to work on this idea, since quite the while.

And it has js about reached the state i can probably get it to MoE arch works, dense models work, cpu/gpu streaming works well, context/kv cache is also dynamically quanted lm_head quants are also there now

The motivation really died out because of this project not getting reviewed its been over a month, leaving this as is, tho its not perfect, but it js about has enough features that i wanted

Here are the final results after a ton of benchmarking-

• Phi-4: ThinTensor 28.21 tok/s vs llama.cpp 17.50 tok/s — 1.61× faster (+61.2%). Quality: 0.999949 cosine vs 0.997281.

• OpenReasoning-Nemotron-14B: ThinTensor 23.79 tok/s vs llama.cpp 19.79 tok/s — 1.20× faster (+20.2%). Quality: 0.999533 vs 0.967140.

• Llama 3.2 11B: ThinTensor 41.02 tok/s vs llama.cpp 39.82 tok/s — 1.03× faster (+3.0%). Quality: 0.995095 vs 0.991938.

• Qwen3.5-9B: ThinTensor 43.28 tok/s vs llama.cpp 42.83 tok/s — 1.01× faster (+1.1%). Quality: 0.973334 vs 0.894908.

• GPT-OSS-20B: ThinTensor 50.34 tok/s vs llama.cpp 40.36 tok/s — 1.25× faster (+24.7%). Quality: 0.983299 vs 0.792350.

*all are apples to apples btw

Overall: ThinTensor beats llama.cpp Q4 throughput on 5/5 tested architectures, with a geometric-mean speedup of approximately 1.203×.

Probably gonna take a break, and learn before comming back to this, after the ship-2 on stardance!

ThinTensor Improvements

• Universal semantic-IR runtime for mixed attention, dense and MoE architectures.
• Per-tensor precision, kernel and residency planning instead of layer-wide quantization.
• Production FP8, INT4, MXFP4, Q2 and Q1 quantizers with calibration-aware scoring.
• Hard ≥0.995 cosine certification with exact routing, greedy-token and KV checks.
• Qwen3.5 hybrid linear-attention, shared-expert and packed rank-3 MoE support.
• Durable grouped-INT4 expert sidecars with bounded-RAM, mmap-based construction.
• Fixed four-matrix Triton projection, cuda/cuda:0, layer-zero and converter-role bugs.
• Improved fused kernels, KV caching, expert routing and decode hot paths.
• Strict cache identity across model, tokenizer, GPU, CUDA, kernels and calibration.
• Remaining goal: prove ≥30 tok/s on Qwen3.5-35B-A3B at certified ≥0.995 cosine.
• Getting way better speed on 27B dense models at a cosine that is also way better

I ammmm tierdd

i tried a billionn things over the last 8 hours i just cant seem to understand why gemma4 doesnt want to workkkkk

I dont have much to write in this devlog, other than, i am not gonna implement gemma arch for now

what I did to fix the numerical drift & get some speedups:

• The Root Cause of the Drift (The Cosine Fix):

I discovered that in triton_kernels.py, the tl.dot_scaled call inside _mxfp_lowbit_selected_tensorcore_matvec_kernel (and the mxfp4 variant) had fast_math=True. The fast_math option drops critical precision bits during accumulator addition which was causing the MoE kernels to completely obliterate the trajectory. I disabled it globally using fast_math=False, and this instantly restored our cosine similarity to a perfect 1.000000 over the entire benchmark trajectory.

• The VRAM Bottleneck: When relying purely on MXFP4 experts, they required over 9.5 GB, breaking the 6.6 GiB budget and forcing the runtime to stream them via PCIe (which tanks performance down to ~5-10tok/s). The Q2 and Q1 low-bit expert decoding kernels actually achieve identical numerical accuracy to the baseline while aggressively packing the weights.
• The 97.87 tok/s Optimization: I configured bench.py and check_baseline_cosine.py to use:

• packed_expert_q1_layers="0:24": All 24 layers of MoE experts are now tightly packed into 1-bit Q1 structures, shrinking the expert memory footprint down to just 2.38 GB (from 9.52 GB).
• dense_int4_layers="all": Shrunk the non-expert dense projection weights (Q, K, V, O) into INT4, dropping their footprint from ~1.1 GB to ~0.27 GB.
• Since the entire model now takes 5.29 GiB (fitting entirely within the 6.6 GiB budget), all layers are pinned directly to VRAM. Because zero layers are streamed (any_streamed=False), the runtime

successfully builds the fully fused cuda_graphs execution path, accelerating generation from a crawl to 97.87 tok/s ON gpt-oss-20b. I believe ill be moving to dense models just about now

All these issues took SOOO many tests to find

What I’ve Done:

1. Fixed KV Cache Indexing in Graph Mode: The previous cudaErrorStreamCaptureUnsupported error and out-of-bounds access were caused by assigning python integers dynamically during capture, and failing to
   respect block sizing. I modified the KV cache’s _in_graph_mode behavior to compute the tensor-based slot offset dynamically ((slot % block_size).view(1)) directly inside the graph capture. This worked
   successfully and fully restored the model’s coherent output without breaking graph logic.
2. Experimented with MLP Block Sizing: I attempted to increase block_m and num_warps in the 1-bit Triton kernel (mxfp_lowbit_selected_swiglu) to reduce redundant memory loads of the intermediate x tensor.
   Unfortunately, register limits and memory layout meant it actually resulted in worse throughput, so I reverted those changes.
3. Experimented with Attention Loop Early Exit: The single_token_gqa_attention_kernel is statically unrolled up to token_bucket (256). I tried adding a dynamic while loop to break early for short context
   sizes, but Triton’s compiler handles static for loops much better.

reached around 65tk/s on gpt-oss-20b, constantly improvising MoE performance, next ima lock in for dense models but for now on MoE

Implemented a huge list of fixes

• Lazy page-ID tensor resolution after quantization

• Storage-alias-aware VRAM accounting

• Correct affine INT4 partial-group quantization

• KV sequence continuity checks

• Free-page geometry invalidation

• Canonical LM-head bias and final-softcap handling

• Certified shortlist logic with exact fallback

• Gemma-style attention scaling/softcap fixes

• Removal of hot-path CUDA scalar tensor allocations

• Device normalization for “cuda”, “cuda:N”, and torch.device

• Profile separation has been reconnected and different profiles select different kernels.

• Paged KV storage exists through SlabbedKVCache.

• A prior paged-attention slowdown was caused by GPU synchronization from .item() calls in SlabbedKVCache.append(), not by paged addressing itself.

• Logical page ownership is now tracked CPU-side to avoid those synchronizations.

• Paged attention has measured approximately:

  • Materialized: 55.1 tok/s
  • Paged: 57.5 tok/s
    on a relevant smaller models

Devlog-idk atp

Well i said that i would stop working on this project last devlog, butttt, i just saw this https://github.com/random-unknown-username/Thintensor/issues/1

Someone tried thintensor on a 128gb unified ram DGX spark, ANDDDDD now i wanna make it good enough like so its a finished thing

Final devlog:

I would be moving from this project to other cool stuff i have in mind, the final benchmarks i got and everything, is pushed, had to do a lot of improvements to get bigger models running just as good as small ones

I know there can be a lot of other optmizations, but maybe a few months later ill come back to this, but for now, its gonna stay as is.

This devlog retains the apples-to-apples GPT-OSS control plus fresh Qwen3.5-9B, Llama 3.2 11B text-path, OpenReasoning-Nemotron-14B, and Phi-4

A LOT of optimizations were done to achive this results and 100s of benchmarks in different profiles and stuff, to get to this final table

ThinRuntime GPT-OSS-20B now achieves:

• 50.3445 tok/s median across three explicit 200-token runs.
• llama.cpp b9964 Q4_K_M: 40.3631 tok/s mean.
• ThinRuntime is 24.73% faster.
• Exact embedding and LM head; Q1/Q2 applies only to experts.
• Zero expert transfers during decode.
• Top-1 unchanged through step 4.
• Cosine: 0.987376 step 1, 0.983299 step 4.
• Accepted one-step configuration: 0.991415 cosine.
• llama.cpp Q4_K_M source-relative cosine: 0.792350.

Worked over a LOT of stuff, like doing apples to apples comparisions of models such as LLama 11b vision instruct, So over time I fixed a lot of stuff

added a new profile, that quants and quants and quants all layers till the model fits your VRAM, it doesnt care if the model is 20gb in weigths and you have a 8gb vram, it will quant and make it fit

So testing this autofit profile on LLama 11b vision instruct, heres what i found

it hit a cosine of 0.939758 and around 24tk/s on a 200 token test

Similarly i tested ollama Llama 11b vision instruct with q4_k_m quant, it got a cosine close to 0.720495 i would say it would be more closer to 0.8 because this cosine was for only top 20 logprobes that ollama exposes

This was done on
8gb vram rtx 5050 laptop
32gb ddr 5
ryzen 7 250
arch linux

and the comparision is apples to apples because the thinruntime also quanted the model aggressively

Devlog: Resolving GPU Memory Corruption & CUDA Assertions on Wide Matvecs

1. The Symptoms

When running inference/benchmarks on unquantized models (like Qwen3.5-9B) using the Triton backend on laptop GPUs (e.g., RTX 5050), the runtime encountered two asynchronous CUDA faults:

• cudaErrorIllegalAddress (Illegal memory access) during cache eviction.
• device-side assert triggered ( indexSelectSmallIndex assertion failed) during embedding vocabulary lookups.

2. The Root Cause (Resource Starvation & Memory Corruption)

• Wide Layer Geometries: In Qwen3.5, the projection matrices (Query, Key, Value) have a columns dimension of cols = 3584 .
• Triton Padded Tiling: Triton pads non-power-of-two shapes to the next power of two ( BLOCK_N = 4096 ).
• Static Tile Size: The default Triton config mapping uses BLOCK_M = 64 for wide matrices. This instructs Triton to compile a kernel loading a massive static tile size of 64 × 4096 = 262,144 elements.
• Register & Shared Memory Overflow: In FP32 accumulation, this tile requires 1 MB of memory per thread block. However, laptop GPU streaming multiprocessors (SMs) have a physical hardware limit of 96KB–100KB
  of shared memory/register files.
• The Failure Cascade: The compiler failed to allocate resources cleanly, resulting in severe register spills and memory read violations. Triton silently returned corrupted/NaN outputs. The subsequent
  argmax logit lookup generated a garbage token index (out of bounds), which triggered the PyTorch device-side assertion on the next step.

3. The Fix (Dynamic Chunked Looping)

To prevent resource starvation, we modified Triton’s launching configuration in triton_kernels.py:

• Wide Column Routing: Added a check to detect if cols > 2048 .
• Looped Execution: Instead of loading the entire 4096 columns block statically, we force the kernel to execute _matvec_looped_kernel using a chunk size of BLOCK_N = 512 .
• Memory Optimization: The kernel loops through the columns sequentially in chunks of 512. This drops the tile size memory requirement per block to under 64 KB, which safely fits inside any laptop GPU’s
  physical SM limits.

4. Results

I am currently running the final profiles ( max-performance and max-max-perf ). Let me check on the status!

Posted about this on reddit!

https://www.reddit.com/r/LocalLLM/comments/1uqr1o3/i_made_a_llm_storage_format_that_makes_llms_run/

Got a lot of feedback, gonna try to work over it over the next few weeks

Final benchmarks

Smaller models have lower cosine because there are like really small like 1B but, some archs are still not fully optimized to run at a good bit faster speed, ik, but i would want to do a ship before working on getting the speed up on those

Finished benchmarking a few more models

Hitting really really good tk/s when compared to hf that even fails to load the models on my 8gb vram, so its a HUGE improvement for gpu poor ppl like me 😭

Reason a lot less improvement for many models is that, each model has a different structure and i havent been able to make a generalized algorithm that would work for all, so for now, we are working with just a simple implementation

Benchmark Methodology and Metrics Calculation

To ensure accurate, robust, and reproducible results, the benchmarking and correctness scoring flow is executed as follows:

1. Benchmark Execution Workflow

Benchmarks are run in isolated, dedicated Python processes for each engine (ThinTensor vs. Hugging Face Transformers) to prevent memory leakage or process interference:

• ThinTensor Baseline:
  • The model is loaded via the thin_runtime.py engine.
  • Weight Fit Planning: The runtime runs _automatic_fit_plan to estimate weight size. If the model’s weights do not fit within the available GPU VRAM (accounting for system/display manager overhead), it dynamically configures streaming weight residency (--weight-residency stream) and streams layers/experts block-by-block from CPU/RAM to the GPU. If the weights fit, it uses fully resident weights (--weight-residency all).
  • Timing: Prefills the prompt (default “Hello”), decodes a number of warmup tokens (default 10) to initialize caches and compile/warm up Triton kernels, and then times the next decode steps (default 200) using wall-clock time (time.perf_counter()).
• Transformers (Hugging Face) Baseline:
  • The model is loaded via a dedicated script bench_hf_transformers_decode.py in BF16 precision.
  • RAM/CPU Offloading: For larger models (e.g. Gemma-4-E2B, OLMoE), loading the model fully on memory-constrained GPUs (like the RTX 5050 Laptop GPU with 7.57 GiB VRAM) causes CUDA OOM. To prevent OOM, RAM offloading is enabled via device_map="auto". Weights that exceed VRAM capacity are kept in CPU system RAM and loaded/swapped as needed.
  • Timing: Like ThinTensor, it pre-runs a prompt prefill, decodes warmup tokens (default 10), and then times steps (default 200) decode steps. The decode loop avoids GPU-to-CPU synchronization (e.g. calling .item() inside the timed loop) to measure pure GPU decoding speed.

2. Metrics & Scores Calculation

• Thin/HF Peak GiB (Peak GPU Memory): Tracks peak allocated VRAM in GiB using torch.cuda.max_memory_allocated(). This measures active tensors allocated by PyTorch’s allocator during the decode phase (excluding unallocated cache reserve or general display server memory).
• Min Cosine (Cosine Similarity): In compare_hf_thin_logits.py, the logits of the final linear projection layer are collected for both models. Cosine similarity is calculated for the target token index at each step. The minimum cosine similarity found across all decode steps (e.g., step 1 and step 10) and prefill lengths (e.g., 1 and 128) is reported.
• Top-1 Match: A boolean check indicating whether the token ID with the highest probability (highest logit value) is identical between ThinTensor and HF at all decode steps.
• Top-5 Set: A boolean check indicating whether the set of top-5 highest-probability token IDs is identical between the two models at all steps (ignoring their internal sorted order).
• Top-5 Order: A boolean check indicating whether the exact sorted order of the top-5 token IDs matches between ThinTensor and HF at all steps.
• KV Retention: Specifies the KV cache precision format (e.g. full BF16).

Dev Log #4 — 91–96 tok/s with Selective MXFP4

This was the first ThinTensor run that felt genuinely fast.

On SmolLM3-3B, the selective MXFP4 gate/up profile reached around:

~91–96 tok/s

and on the strong tested record it still preserved:

cosine_similarity: 0.9994624257087708
top1_same: true
top5_overlap: 0.8
generated_token_match_rate: 0.875
KV retention: full BF16

That cosine is extremely close to the HF BF16 reference:

1.0 - 0.9994624257087708 = 0.0005375742912292

So the run was only about 0.054% away from perfect cosine on that tested checkpoint.

Why this was not just normal quantization

The profile was selective:

MXFP4 gate/up layers: 0:24
MXFP4 row postscale: enabled
LM head: FP8 guarded path
KV cache: full BF16
attention path: causal KV
QKV/O/down: not blindly pushed into MXFP4

A full MXFP4 sweep over QKV, O, down, and gate/up was fast, but it destroyed the logits. That proved the obvious thing: fewer bytes alone is not enough.

ThinTensor’s win comes from being role-aware.

The runtime knows what each tensor does:

Q / K / V
O projection
gate projection
up projection
down projection
norm
embedding
LM head
KV cache

That means the runtime can apply aggressive precision only to the parts that tolerate it.

For this result, the safe target was the MLP gate/up path.

Why gate/up was the right target

Single-token decoding is mostly a weight-read problem.

Tthe goal became:

reduce gate/up bytes
keep attention stable
keep KV exact
keep LM head guarded
preserve token ranking

That is exactly what the selective MXFP4 profile does.

It uses MXFP4 on the gate/up projections for layers 0:24, but it does not blindly lower every sensitive path.

Where Blackwell helps

The test GPU is an RTX 5050 laptop GPU, which is Blackwell.

That matters because Blackwell has native low-precision paths for FP4/MXFP4-style execution. ThinTensor uses this through Triton kernels that treat MXFP4 as an execution format, not just a smaller storage format.

The idea is:

store/read fewer bytes
use block scales
multiply against BF16 activations
accumulate safely
then apply row postscale

The goal is to make each token cheaper to decode while keeping the output distribution close to HF BF16.

The code side

The relevant code is built around these ideas:

1. Honest quantization descriptors

The QuantizationDescriptor describes the source format:

method
storage_dtype
bits
group_size
scale_dtype
zero_point
modules_to_not_convert
source_config

For example:

INT4 != FP4
Q8 != FP8
MXFP4 != generic Q4

The precision_ladder() function makes that explicit. It lists safe choices, opt-in choices, and lossy choices.

So instead of silently lowering precision, the runtime can say:

preserve native format
fallback to BF16
or require explicit validation before requantization

2. Execution planning

plan_quantization() decides what should happen at runtime.

It can choose:

preserve source storage
use native kernel
dequantize to BF16
or reject unsupported requantization

The important flag is this:

exact_storage_preserved

If this is true, the runtime is using the source encoding directly.

Whats next

I would have to implement all mainstream model archs manually because all are diff, and like right now you can see, there are a lot of models that are heavily un-optimized

Also im messaging stardance team on slack these devlogs too smol for me

Just hit 91tk/s with “cosine_similarity”: 0.9994624257087708

its actually crazy its like 5x faster the hf full devlog comming soon!

ITS LIKE 0.054% away from perfect!!!

The selective MXFP4 gate/up profile reached ~96 tok/s while preserving top-1 on the tested SmolLM3-3B through my thintensor runtimeeeee

Full DevLogs gonna be crazy

Initial benchmarks based on some testing on a 3B model

Why response similarity?

Because we are making a storage format not “just” a runtime, so we need to ensure the model we are optimizing using thintensor runtime it HAS to be replying with the same responses like the original model!

So with casual Kv implemented these are the benchmarks i gathered, if you wanna learn what quality fp8 is read my last devlog :D

PS- the vram usage on our ThinTensor quality FP8 is just 500 mb more than the q8 even though there is a HUGE difference in the response similarity

Dev Log #3 — SmolLM3-3B Results

SmolLM3-3B was the first real test of ThinTensor as an inference runtime.

The baseline was Hugging Face BF16 on my RTX 5050 laptop GPU:

HF BF16: ~17.8 tok/s

ThinTensor BF16 reached:

ThinTensor BF16: ~32–33 tok/s

with cosine around 0.999, meaning the logits stayed extremely close to the original SafeTensors/HF path. That matters because a storage/runtime format should not make the model noticeably worse.

Then came the quality FP8 profile:

ThinTensor quality FP8: ~46–48 tok/s

This was not “quantize everything.” The profile was selective:

gate/up projections: FP8
middle down projections: FP8
attention path: BF16
O-proj: BF16
LM head: BF16
KV cache: BF16

ThinTensor can do this because the .thin manifest and execution tape know tensor roles: Q, K, V, O, gate, up, down, norm, embedding, LM head, etc.

The runtime can then lower precision only where the model tolerates it, while keeping sensitive paths in BF16.

The kernel side reflects this. Attention uses single-token GQA and maps each query head to the right KV head:

head = tl.program_id(0)
kv_head = head // (heads // kv_heads)

scores = tl.sum(key * q[None, :], axis=1) * scale

The FP8 MLP path uses scaled matvecs:

w = tl.load(weight + offs_m[:, None] * stride_wm + offs_n[None, :])
xv = tl.load(x + offs_n).to(tl.float32)
scale = tl.load(scales + offs_m).to(tl.float32)

acc = tl.sum(w * xv[None, :], axis=1) * scale

In one full-vocab centered-logit comparison:

GGUF Q8_0 min cosine:              0.952666
ThinTensor BF16 min cosine:        0.999596
ThinTensor quality FP8 min cosine: 0.995366

So ThinTensor quality FP8 stayed much closer to BF16 than the measured Q8 path, while staying in the same speed class.

Final speed table:

HF BF16:                 ~17.8 tok/s
ThinTensor BF16:         ~32–33 tok/s
ThinTensor quality FP8:  ~46–48 tok/s
Ollama / GGUF Q8_0:      ~44.1 tok/s

Headline:

ThinTensor BF16 was ~1.8x faster than HF BF16.
ThinTensor quality FP8 was ~2.6x faster than HF BF16.
ThinTensor quality FP8 was faster than the measured Q8 baseline while preserving much better logit fidelity.

This is where ThinTensor stopped being just an archive experiment and started looking like a real inference runtime :D

Js got to know that devlogs longer than 10hrs are stripped to 10 😭, i cant have more than 4k char so bear with me theres a LOT to tell that im not able to fit, but ill try my best anyways

Dev Log #2 — Converting SafeTensors Into .thin

(had to cut so much explaination bcs i HATE THE CHAR LIMIT, id add them in another devlog)

The main conversion flow is:

pub fn convert_hf(options: ConvertHfOptions) -> Result<ConvertHfResult> {
    let config: Value = serde_json::from_reader(
        File::open(options.hf_dir.join("config.json"))?
    )?;

    let safetensor_paths = collect_safetensors(&options.hf_dir)?;
    let tensors = collect_tensors(&safetensor_paths)?;

    let layers = required_u32(&config, "num_hidden_layers")?;
    let model = build_model_spec(&config, layers, &tensors, &options, &mut warnings)?;
    let pages = build_pages(&tensors, &config)?;
    let execution_tape = build_execution_tape(layers, &tensors, &mut warnings)?;

    let manifest = Manifest {
        format: FORMAT_NAME.to_string(),
        version: FORMAT_VERSION,
        model,
        execution_tape,
        pages,
        memory_plan: build_memory_plan(&config, &tensors),
    };

    write_archive_pages(&options.out_path, &manifest, &archive_pages)?;
    let archive = Archive::open(&options.out_path)?;
    verify_archive(&archive)?;

    Ok(ConvertHfResult { archive, warnings })
}

The converter memory-maps each SafeTensors file read-only:

let mmap = unsafe { Mmap::map(&file)? };
let safetensors = SafeTensors::deserialize(&mmap)?;
let base = mmap.as_ptr() as usize;

Then every tensor becomes a page candidate. The converter records where the tensor bytes live inside the mapped file:

let data = tensor.data();
let offset = (data.as_ptr() as usize).checked_sub(base)?;

and stores:

struct TensorPage {
    id: String,
    path: PathBuf,
    offset: u64,
    size: u64,
    checksum: [u8; 32],
    dtype: String,
    shape: Vec<u64>,
}

This avoids rebuilding every tensor in memory. The archive can later copy the original byte range directly:

ArchivePageSource::FileRange {
    path: tensor.path.clone(),
    offset: tensor.offset,
}

After collecting raw tensor data, the converter builds the model spec. It extracts or infers things like hidden size, layer count, attention heads, KV heads, head dimension, RoPE settings, vocab size, activation type, quantization config, and required runtime operators.

One important field is attention type:

let attention_kind = if kv_heads == heads {
    "mha"
} else if kv_heads == 1 {
    "mqa"
} else {
    "gqa"
};

This matters because many modern decoder models use grouped-query attention. If the runtime only sees shapes but does not understand Q heads vs KV heads, causal attention can be wrong while still looking structurally valid.

The converter also records required operators:

let mut required_operators = vec![
    "embedding".to_string(),
    "rms_norm".to_string(),
    "qkv_projection".to_string(),
    "rope".to_string(),
    format!("{attention_kind}_attention"),
    "o_projection".to_string(),
    "lm_head".to_string(),
];

For dense MLPs it adds:

gated_activation
down_projection

For MoE models it records router and expert stages instead.

the rest would be explained in another devlog, i hate the char limit

(had to re-write like 10 times btw)

Dev Log #1 — Why ThinTensor Exists

SafeTensors is already a very good format for what it was designed to do: store tensors safely. It keeps tensor metadata simple. For normal model distribution, that is enough.

The problem starts when the tensor file becomes the runtime boundary.

That is enough if another framework already knows the model class and execution graph. But for an inference runtime, especially one focused on single-token decode, those fields are not enough. The runtime needs to know what each tensor means.

The .thin Format

The first .thin archive is intentionally simple:

.thin = fixed header + JSON manifest + page table + raw page data

The Rust implementation defines the archive identity and header size directly:

pub const MAGIC: &[u8; 8] = b"THINv0\0\0";
pub const HEADER_LEN: u32 = 88;

The header stores the physical structure of the archive:

pub struct Header {
    pub header_len: u32,
    pub version: u32,
    pub manifest_off: u64,
    pub manifest_len: u64,
    pub page_table_off: u64,
    pub page_count: u64,
    pub data_off: u64,
    pub archive_hash: [u8; 32],
}

This lets the loader open the file without guessing. It reads the header, jumps to the manifest, then reads the page table, then maps the raw page data.

The manifest is JSON because the early format needed to stay inspectable. It contains the model-level information needed to reconstruct decode:

architecture
hidden size
number of layers
attention heads
KV heads
head dimension
vocabulary size
RoPE settings
tensor names
tensor shapes
tensor dtypes
tensor roles
quantization metadata
runtime/backend hints
validation metadata

The page table connects the manifest to real bytes inside the archive:

pub struct PageTableRecord {
    pub page_id: String,
    pub offset: u64,
    pub stored_size: u64,
    pub raw_size: u64,
    pub flags: u32,
    pub checksum: [u8; 32],
}

A page record gives the runtime a verified byte range, not just a tensor name. This matters because pages can later be scheduled, streamed, cached, kept on CPU, moved to GPU, or assigned different execution policies.

Packing and Verification

The packer does not blindly write files into an archive. It validates the manifest first:

let report = validate_manifest(&manifest);
if !report.is_ok() {
    bail!("manifest invalid:\n{}", report.errors.join("\n"));
}

Then every page declared in the manifest must exist and match its declared size:

if bytes.len() as u64 != page.size {
    bail!("page declared size does not match file size");
}

It also verifies the page checksum:

let checksum = blake3::hash(&bytes);
if checksum.as_bytes().as_slice() != expected.as_slice() {
    bail!("page checksum does not match manifest");
}

Only after validation does the writer emit the archive:

write_header(&mut file, &header)?;
file.write_all(&manifest_bytes)?;
for record in &records {
    write_page_table_record(&mut file, record)?;
}
for page in pages {
    write_page_source(&mut file, page)?;
}

So the final file is structured as:

[ header ]
[ manifest JSON ]
[ page table records ]
[ raw page blobs ]