I ammmm tierdd

i tried a billionn things over the last 8 hours i just cant seem to understand why gemma4 doesnt want to workkkkk

I dont have much to write in this devlog, other than, i am not gonna implement gemma arch for now

what I did to fix the numerical drift & get some speedups:

• The Root Cause of the Drift (The Cosine Fix):

I discovered that in triton_kernels.py, the tl.dot_scaled call inside _mxfp_lowbit_selected_tensorcore_matvec_kernel (and the mxfp4 variant) had fast_math=True. The fast_math option drops critical precision bits during accumulator addition which was causing the MoE kernels to completely obliterate the trajectory. I disabled it globally using fast_math=False, and this instantly restored our cosine similarity to a perfect 1.000000 over the entire benchmark trajectory.

• The VRAM Bottleneck: When relying purely on MXFP4 experts, they required over 9.5 GB, breaking the 6.6 GiB budget and forcing the runtime to stream them via PCIe (which tanks performance down to ~5-10tok/s). The Q2 and Q1 low-bit expert decoding kernels actually achieve identical numerical accuracy to the baseline while aggressively packing the weights.
• The 97.87 tok/s Optimization: I configured bench.py and check_baseline_cosine.py to use:

• packed_expert_q1_layers="0:24": All 24 layers of MoE experts are now tightly packed into 1-bit Q1 structures, shrinking the expert memory footprint down to just 2.38 GB (from 9.52 GB).
• dense_int4_layers="all": Shrunk the non-expert dense projection weights (Q, K, V, O) into INT4, dropping their footprint from ~1.1 GB to ~0.27 GB.
• Since the entire model now takes 5.29 GiB (fitting entirely within the 6.6 GiB budget), all layers are pinned directly to VRAM. Because zero layers are streamed (any_streamed=False), the runtime

successfully builds the fully fused cuda_graphs execution path, accelerating generation from a crawl to 97.87 tok/s ON gpt-oss-20b. I believe ill be moving to dense models just about now

All these issues took SOOO many tests to find

What I’ve Done:

1. Fixed KV Cache Indexing in Graph Mode: The previous cudaErrorStreamCaptureUnsupported error and out-of-bounds access were caused by assigning python integers dynamically during capture, and failing to
   respect block sizing. I modified the KV cache’s _in_graph_mode behavior to compute the tensor-based slot offset dynamically ((slot % block_size).view(1)) directly inside the graph capture. This worked
   successfully and fully restored the model’s coherent output without breaking graph logic.
2. Experimented with MLP Block Sizing: I attempted to increase block_m and num_warps in the 1-bit Triton kernel (mxfp_lowbit_selected_swiglu) to reduce redundant memory loads of the intermediate x tensor.
   Unfortunately, register limits and memory layout meant it actually resulted in worse throughput, so I reverted those changes.
3. Experimented with Attention Loop Early Exit: The single_token_gqa_attention_kernel is statically unrolled up to token_bucket (256). I tried adding a dynamic while loop to break early for short context
   sizes, but Triton’s compiler handles static for loops much better.

reached around 65tk/s on gpt-oss-20b, constantly improvising MoE performance, next ima lock in for dense models but for now on MoE

Implemented a huge list of fixes

• Lazy page-ID tensor resolution after quantization

• Storage-alias-aware VRAM accounting

• Correct affine INT4 partial-group quantization

• KV sequence continuity checks

• Free-page geometry invalidation

• Canonical LM-head bias and final-softcap handling

• Certified shortlist logic with exact fallback

• Gemma-style attention scaling/softcap fixes

• Removal of hot-path CUDA scalar tensor allocations

• Device normalization for “cuda”, “cuda:N”, and torch.device

• Profile separation has been reconnected and different profiles select different kernels.

• Paged KV storage exists through SlabbedKVCache.

• A prior paged-attention slowdown was caused by GPU synchronization from .item() calls in SlabbedKVCache.append(), not by paged addressing itself.

• Logical page ownership is now tracked CPU-side to avoid those synchronizations.

• Paged attention has measured approximately:

  • Materialized: 55.1 tok/s
  • Paged: 57.5 tok/s
    on a relevant smaller models

Devlog-idk atp

Well i said that i would stop working on this project last devlog, butttt, i just saw this https://github.com/random-unknown-username/Thintensor/issues/1

Someone tried thintensor on a 128gb unified ram DGX spark, ANDDDDD now i wanna make it good enough like so its a finished thing

the website is more complete now

DevLog-3

Placement done, few things-

• back side has a few caps and stuff, because the space is tight
• the board is 22x22 rn, moving to a 25x25 soon, to make the board’s back completely empty and keeping all caps and stuff at the top!

Devlog-2

Completed the SoC design, now on to placement, a few things-

• The decoupling is just perfect to make the IC run and not burn
• Minimal stuff is used because of area constrains i plan to have
• Some GPIOs wont be exposed to SoM pins

Devlog-1:

Making the canva for video games!!!

The Tech Stack

I wanted this engine to run natively in the browser without massive downloads, so I chose:

• React + Vite for the frontend UI and state management.

• Three.js & React-Three-Fiber (R3F) for the 3D rendering pipeline.

• Cannon-es (@react-three/cannon) for rigid-body physics.

• Zustand for lightweight, blazing-fast state management.

• React Flow (@xyflow/react) for the Visual Scripting node editor.

  1. Building the ECS (Entity-Component System) in Zustand

Most React apps map data to UI components linearly, but game engines need an Entity-Component System (ECS) to handle hundreds of objects dynamically. I built a custom ECS using Zustand.

    // The core engine state powering the ECS
    interface EngineState {
      entities: Record<string, Entity>;
      addEntity: (name: string) => string;
      addComponent: (entityId: string, componentType: ComponentType) => void;
      updateComponent: (entityId: string, componentType: ComponentType, data: any) => void;
    }

This allows the engine’s Outliner and Inspector panels to dynamically edit any object in the scene in real-time, just like Unity or Unreal.

2. The Visual Scripting Compiler

The coolest feature I got working today was the Visual Script Editor. I used React Flow to let users connect nodes (like “On Update” -> “Move To Player”). But how does a node graph become executable game?

I wrote a compiler that traverses the node graph and translates it into a raw JavaScript string. This string is saved into the entity’s script component and executed dynamically in the game loop using a
Function constructor!

    // A snippet of the compiler output injected into the game loop
    } else if (node.data.label === 'Enemy: Move To Player') {
      output += `
        if (api.velocity) {
          const toPlayer = camera.position.clone().sub(mesh.position);
          toPlayer.y = 0;
          toPlayer.normalize();
          api.velocity.set(toPlayer.x * 4, -1, toPlayer.z * 4);
        }
      `;
    }

3. Bypassing React for Performance

The biggest hurdle today was the physics loop. Initially, the enemy tracking script updated the Zustand state every frame. With 15 enemies running at 60fps, it was triggering 900 React state updates a second, completely freezing the browser!

The novel solution for this was bypassing React’s render lifecycle entirely for physics. Instead of updating the state.entities transform data, the compiled scripts now directly tap into the raw Cannon-es physics
API (api.velocity.set()), mutating the physics hull natively. The performance skyrocketed from a frozen screen to a buttery smooth 60fps.

Devlog-1

Made the Power circuit with 3 JW5211 ICs each do-

• 5v -> 3.3v
• 5v -> 2.5v (closer to 2.6v for ddr1)
• 5v -> 1.1v

Required for the F1C100s from allwinner!
a small linux capable SoC

Also added a SPI flash and some status LEDs and stuff

Final devlog:

I would be moving from this project to other cool stuff i have in mind, the final benchmarks i got and everything, is pushed, had to do a lot of improvements to get bigger models running just as good as small ones

I know there can be a lot of other optmizations, but maybe a few months later ill come back to this, but for now, its gonna stay as is.

Devlog

this was originally supposed to be shipped to clubsite ysws but that is unfortunately no longer active, so I am just adding it here

A few cool things in the website-

• Playable games that were made by members (the raycaster demo was made by me for hackclub sprig!!!)
• this site would be under constant change for the next few days, we are finalizing a design with out school administration
• All animations were coded by me :D
• The logo and all art and stuff was design by other co-leaders from the club!

I figured I’ll just add the time i spent on stardance because the older clubsite ysws is inactive now

This devlog retains the apples-to-apples GPT-OSS control plus fresh Qwen3.5-9B, Llama 3.2 11B text-path, OpenReasoning-Nemotron-14B, and Phi-4

A LOT of optimizations were done to achive this results and 100s of benchmarks in different profiles and stuff, to get to this final table

ThinRuntime GPT-OSS-20B now achieves:

• 50.3445 tok/s median across three explicit 200-token runs.
• llama.cpp b9964 Q4_K_M: 40.3631 tok/s mean.
• ThinRuntime is 24.73% faster.
• Exact embedding and LM head; Q1/Q2 applies only to experts.
• Zero expert transfers during decode.
• Top-1 unchanged through step 4.
• Cosine: 0.987376 step 1, 0.983299 step 4.
• Accepted one-step configuration: 0.991415 cosine.
• llama.cpp Q4_K_M source-relative cosine: 0.792350.

Worked over a LOT of stuff, like doing apples to apples comparisions of models such as LLama 11b vision instruct, So over time I fixed a lot of stuff

added a new profile, that quants and quants and quants all layers till the model fits your VRAM, it doesnt care if the model is 20gb in weigths and you have a 8gb vram, it will quant and make it fit

So testing this autofit profile on LLama 11b vision instruct, heres what i found

it hit a cosine of 0.939758 and around 24tk/s on a 200 token test

Similarly i tested ollama Llama 11b vision instruct with q4_k_m quant, it got a cosine close to 0.720495 i would say it would be more closer to 0.8 because this cosine was for only top 20 logprobes that ollama exposes

This was done on
8gb vram rtx 5050 laptop
32gb ddr 5
ryzen 7 250
arch linux

and the comparision is apples to apples because the thinruntime also quanted the model aggressively

Devlog: Resolving GPU Memory Corruption & CUDA Assertions on Wide Matvecs

1. The Symptoms

When running inference/benchmarks on unquantized models (like Qwen3.5-9B) using the Triton backend on laptop GPUs (e.g., RTX 5050), the runtime encountered two asynchronous CUDA faults:

• cudaErrorIllegalAddress (Illegal memory access) during cache eviction.
• device-side assert triggered ( indexSelectSmallIndex assertion failed) during embedding vocabulary lookups.

2. The Root Cause (Resource Starvation & Memory Corruption)

• Wide Layer Geometries: In Qwen3.5, the projection matrices (Query, Key, Value) have a columns dimension of cols = 3584 .
• Triton Padded Tiling: Triton pads non-power-of-two shapes to the next power of two ( BLOCK_N = 4096 ).
• Static Tile Size: The default Triton config mapping uses BLOCK_M = 64 for wide matrices. This instructs Triton to compile a kernel loading a massive static tile size of 64 × 4096 = 262,144 elements.
• Register & Shared Memory Overflow: In FP32 accumulation, this tile requires 1 MB of memory per thread block. However, laptop GPU streaming multiprocessors (SMs) have a physical hardware limit of 96KB–100KB
  of shared memory/register files.
• The Failure Cascade: The compiler failed to allocate resources cleanly, resulting in severe register spills and memory read violations. Triton silently returned corrupted/NaN outputs. The subsequent
  argmax logit lookup generated a garbage token index (out of bounds), which triggered the PyTorch device-side assertion on the next step.

3. The Fix (Dynamic Chunked Looping)

To prevent resource starvation, we modified Triton’s launching configuration in triton_kernels.py:

• Wide Column Routing: Added a check to detect if cols > 2048 .
• Looped Execution: Instead of loading the entire 4096 columns block statically, we force the kernel to execute _matvec_looped_kernel using a chunk size of BLOCK_N = 512 .
• Memory Optimization: The kernel loops through the columns sequentially in chunks of 512. This drops the tile size memory requirement per block to under 64 KB, which safely fits inside any laptop GPU’s
  physical SM limits.

4. Results

I am currently running the final profiles ( max-performance and max-max-perf ). Let me check on the status!

Posted about this on reddit!

https://www.reddit.com/r/LocalLLM/comments/1uqr1o3/i_made_a_llm_storage_format_that_makes_llms_run/

Got a lot of feedback, gonna try to work over it over the next few weeks

Final benchmarks

Smaller models have lower cosine because there are like really small like 1B but, some archs are still not fully optimized to run at a good bit faster speed, ik, but i would want to do a ship before working on getting the speed up on those