Finished benchmarking a few more models

Hitting really really good tk/s when compared to hf that even fails to load the models on my 8gb vram, so its a HUGE improvement for gpu poor ppl like me 😭

Reason a lot less improvement for many models is that, each model has a different structure and i havent been able to make a generalized algorithm that would work for all, so for now, we are working with just a simple implementation

Benchmark Methodology and Metrics Calculation

To ensure accurate, robust, and reproducible results, the benchmarking and correctness scoring flow is executed as follows:

1. Benchmark Execution Workflow

Benchmarks are run in isolated, dedicated Python processes for each engine (ThinTensor vs. Hugging Face Transformers) to prevent memory leakage or process interference:

• ThinTensor Baseline:
  • The model is loaded via the thin_runtime.py engine.
  • Weight Fit Planning: The runtime runs _automatic_fit_plan to estimate weight size. If the model’s weights do not fit within the available GPU VRAM (accounting for system/display manager overhead), it dynamically configures streaming weight residency (--weight-residency stream) and streams layers/experts block-by-block from CPU/RAM to the GPU. If the weights fit, it uses fully resident weights (--weight-residency all).
  • Timing: Prefills the prompt (default “Hello”), decodes a number of warmup tokens (default 10) to initialize caches and compile/warm up Triton kernels, and then times the next decode steps (default 200) using wall-clock time (time.perf_counter()).
• Transformers (Hugging Face) Baseline:
  • The model is loaded via a dedicated script bench_hf_transformers_decode.py in BF16 precision.
  • RAM/CPU Offloading: For larger models (e.g. Gemma-4-E2B, OLMoE), loading the model fully on memory-constrained GPUs (like the RTX 5050 Laptop GPU with 7.57 GiB VRAM) causes CUDA OOM. To prevent OOM, RAM offloading is enabled via device_map="auto". Weights that exceed VRAM capacity are kept in CPU system RAM and loaded/swapped as needed.
  • Timing: Like ThinTensor, it pre-runs a prompt prefill, decodes warmup tokens (default 10), and then times steps (default 200) decode steps. The decode loop avoids GPU-to-CPU synchronization (e.g. calling .item() inside the timed loop) to measure pure GPU decoding speed.

2. Metrics & Scores Calculation

• Thin/HF Peak GiB (Peak GPU Memory): Tracks peak allocated VRAM in GiB using torch.cuda.max_memory_allocated(). This measures active tensors allocated by PyTorch’s allocator during the decode phase (excluding unallocated cache reserve or general display server memory).
• Min Cosine (Cosine Similarity): In compare_hf_thin_logits.py, the logits of the final linear projection layer are collected for both models. Cosine similarity is calculated for the target token index at each step. The minimum cosine similarity found across all decode steps (e.g., step 1 and step 10) and prefill lengths (e.g., 1 and 128) is reported.
• Top-1 Match: A boolean check indicating whether the token ID with the highest probability (highest logit value) is identical between ThinTensor and HF at all decode steps.
• Top-5 Set: A boolean check indicating whether the set of top-5 highest-probability token IDs is identical between the two models at all steps (ignoring their internal sorted order).
• Top-5 Order: A boolean check indicating whether the exact sorted order of the top-5 token IDs matches between ThinTensor and HF at all steps.
• KV Retention: Specifies the KV cache precision format (e.g. full BF16).