Hitting 100hrs and final ship

Didnt get time to work on this idea, since quite the while.

And it has js about reached the state i can probably get it to MoE arch works, dense models work, cpu/gpu streaming works well, context/kv cache is also dynamically quanted lm_head quants are also there now

The motivation really died out because of this project not getting reviewed its been over a month, leaving this as is, tho its not perfect, but it js about has enough features that i wanted

Here are the final results after a ton of benchmarking-

• Phi-4: ThinTensor 28.21 tok/s vs llama.cpp 17.50 tok/s — 1.61× faster (+61.2%). Quality: 0.999949 cosine vs 0.997281.

• OpenReasoning-Nemotron-14B: ThinTensor 23.79 tok/s vs llama.cpp 19.79 tok/s — 1.20× faster (+20.2%). Quality: 0.999533 vs 0.967140.

• Llama 3.2 11B: ThinTensor 41.02 tok/s vs llama.cpp 39.82 tok/s — 1.03× faster (+3.0%). Quality: 0.995095 vs 0.991938.

• Qwen3.5-9B: ThinTensor 43.28 tok/s vs llama.cpp 42.83 tok/s — 1.01× faster (+1.1%). Quality: 0.973334 vs 0.894908.

• GPT-OSS-20B: ThinTensor 50.34 tok/s vs llama.cpp 40.36 tok/s — 1.25× faster (+24.7%). Quality: 0.983299 vs 0.792350.

*all are apples to apples btw

Overall: ThinTensor beats llama.cpp Q4 throughput on 5/5 tested architectures, with a geometric-mean speedup of approximately 1.203×.

Probably gonna take a break, and learn before comming back to this, after the ship-2 on stardance!