Devlog 2

Built and validated a real native Vulkan RAYPAGE path on the RTX 5050. RT cores perform BVH traversal and page candidate pruning entirely on-device, with no CPU routing, readback, per-token BVH rebuild, or host synchronization.

What passed:

• Native RT integration and live router execution.
• BVH rejection: ~99.57%.
• Full-model deterministic chat execution.
• Fidelity: final-logit cosine ~0.9964 and transformed hidden cosine >0.998.
• 14B Qwen2.5-Q2_K model loads, chats, stays deterministic, and fits within 8 GB VRAM (~5.64 GB peak).

What failed:

• End-to-end speed: native RT Qwen3 reached ~88 tok/s versus dense CUDA ~362 tok/s.
• Software sparse was also slower (~115 tok/s), proving routing was not the main issue.
• Precomputed perfect page IDs still achieved only ~106 tok/s, eliminating RT traversal as the bottleneck.
• Indirect quantized MUL_MAT_ID was ~5.15 ms versus ~0.75 ms dense: ~6.9× slower.
• Useful-page recall was ~98.69%, below the 99.5% gate.
• 14B page-major execution reached only ~0.5× dense.
• Fine-grained arbitrary pages destroy contiguous reads, tensor-core tiling, warp coherence, and fused-kernel efficiency.

New work:

• Implemented quantization-safe coarse 128-channel superblocks.
• Built a 6-block trunk + 18-page Q8_0 model with contiguous page-major storage.
• Added prefix slicing so active bundle counts can use ordinary contiguous MUL_MAT kernels without rebuilding artifacts.
• Pulled CUTLASS and confirmed explicit SM120 NVFP4/BF16 GEMM examples exist.
• Initial superblock active-1 fast path was ~68 tok/s, so it has not yet solved the problem.

Most promising directions:

1. RT selects 1–2 coarse bundles, then one fused contiguous kernel executes gate/up/SwiGLU/down.
2. Native block-scaled NVFP4/FP4 kernels, avoiding FP4→BF16 materialization.
3. Long-context KV-page pruning, where avoided attention work is much larger than FFN routing overhead.
4. Hybrid bundle templates: fewer fixed layouts, but optimized for real decode shapes.

Current conclusion: RT pruning works; arbitrary sparse FFN execution is the failure.

MIRAGE Devlog 1 - First 10 Hours

• Audited Qwen3-Coder-30B-A3B. Direct conversion was rejected: its experts need about 54 GiB in BF16, while the artifact limit is 7.2 GiB.

• Built the MIRAGE-7B budget solver and preserved all capture, replay, training, Vulkan, CUDA, and benchmark work.

• Built a real 97M prototype with attention, RoPE, KV cache, shared experts, hard routing, and text generation.

• Training, checkpointing, routing, and tiny-set memorization worked. File/repository splits and exact/near-duplicate checks were added, but generalization is unproven.

• The 15,000-step run stayed finite and passed checkpoint round-trip checks, but quality remained experimental.

• Vectorized bundles were 13.82x faster in isolation. Native Vulkan ray tracing worked, but was 5.17x slower than software and rejected no candidates.- Added frozen search schemas, task/evidence graphs, replay, and untrusted-page handling. Started audited data builders and Qwen3-0.6B upcycling; learning and live web remain unfinished.

Made the initial UI and stuff

Looks really cool, manually made, tho AI was used to assist me in copying CSS from https://factory.ai

just the landing page is made for now, with a few animations, the most time taking part was writing down the whole ideas and thinking through the whole arch to like write everything on the landing page

ThinTensor Improvements

• Universal semantic-IR runtime for mixed attention, dense and MoE architectures.
• Per-tensor precision, kernel and residency planning instead of layer-wide quantization.
• Production FP8, INT4, MXFP4, Q2 and Q1 quantizers with calibration-aware scoring.
• Hard ≥0.995 cosine certification with exact routing, greedy-token and KV checks.
• Qwen3.5 hybrid linear-attention, shared-expert and packed rank-3 MoE support.
• Durable grouped-INT4 expert sidecars with bounded-RAM, mmap-based construction.
• Fixed four-matrix Triton projection, cuda/cuda:0, layer-zero and converter-role bugs.
• Improved fused kernels, KV caching, expert routing and decode hot paths.
• Strict cache identity across model, tokenizer, GPU, CUDA, kernels and calibration.
• Remaining goal: prove ≥30 tok/s on Qwen3.5-35B-A3B at certified ≥0.995 cosine.
• Getting way better speed on 27B dense models at a cosine that is also way better

I ammmm tierdd

i tried a billionn things over the last 8 hours i just cant seem to understand why gemma4 doesnt want to workkkkk

I dont have much to write in this devlog, other than, i am not gonna implement gemma arch for now

what I did to fix the numerical drift & get some speedups:

• The Root Cause of the Drift (The Cosine Fix):

I discovered that in triton_kernels.py, the tl.dot_scaled call inside _mxfp_lowbit_selected_tensorcore_matvec_kernel (and the mxfp4 variant) had fast_math=True. The fast_math option drops critical precision bits during accumulator addition which was causing the MoE kernels to completely obliterate the trajectory. I disabled it globally using fast_math=False, and this instantly restored our cosine similarity to a perfect 1.000000 over the entire benchmark trajectory.

• The VRAM Bottleneck: When relying purely on MXFP4 experts, they required over 9.5 GB, breaking the 6.6 GiB budget and forcing the runtime to stream them via PCIe (which tanks performance down to ~5-10tok/s). The Q2 and Q1 low-bit expert decoding kernels actually achieve identical numerical accuracy to the baseline while aggressively packing the weights.
• The 97.87 tok/s Optimization: I configured bench.py and check_baseline_cosine.py to use:

• packed_expert_q1_layers="0:24": All 24 layers of MoE experts are now tightly packed into 1-bit Q1 structures, shrinking the expert memory footprint down to just 2.38 GB (from 9.52 GB).
• dense_int4_layers="all": Shrunk the non-expert dense projection weights (Q, K, V, O) into INT4, dropping their footprint from ~1.1 GB to ~0.27 GB.
• Since the entire model now takes 5.29 GiB (fitting entirely within the 6.6 GiB budget), all layers are pinned directly to VRAM. Because zero layers are streamed (any_streamed=False), the runtime

successfully builds the fully fused cuda_graphs execution path, accelerating generation from a crawl to 97.87 tok/s ON gpt-oss-20b. I believe ill be moving to dense models just about now

All these issues took SOOO many tests to find

What I’ve Done:

1. Fixed KV Cache Indexing in Graph Mode: The previous cudaErrorStreamCaptureUnsupported error and out-of-bounds access were caused by assigning python integers dynamically during capture, and failing to
   respect block sizing. I modified the KV cache’s _in_graph_mode behavior to compute the tensor-based slot offset dynamically ((slot % block_size).view(1)) directly inside the graph capture. This worked
   successfully and fully restored the model’s coherent output without breaking graph logic.
2. Experimented with MLP Block Sizing: I attempted to increase block_m and num_warps in the 1-bit Triton kernel (mxfp_lowbit_selected_swiglu) to reduce redundant memory loads of the intermediate x tensor.
   Unfortunately, register limits and memory layout meant it actually resulted in worse throughput, so I reverted those changes.
3. Experimented with Attention Loop Early Exit: The single_token_gqa_attention_kernel is statically unrolled up to token_bucket (256). I tried adding a dynamic while loop to break early for short context
   sizes, but Triton’s compiler handles static for loops much better.

reached around 65tk/s on gpt-oss-20b, constantly improvising MoE performance, next ima lock in for dense models but for now on MoE

Implemented a huge list of fixes

• Lazy page-ID tensor resolution after quantization

• Storage-alias-aware VRAM accounting

• Correct affine INT4 partial-group quantization

• KV sequence continuity checks

• Free-page geometry invalidation

• Canonical LM-head bias and final-softcap handling

• Certified shortlist logic with exact fallback

• Gemma-style attention scaling/softcap fixes

• Removal of hot-path CUDA scalar tensor allocations

• Device normalization for “cuda”, “cuda:N”, and torch.device

• Profile separation has been reconnected and different profiles select different kernels.

• Paged KV storage exists through SlabbedKVCache.

• A prior paged-attention slowdown was caused by GPU synchronization from .item() calls in SlabbedKVCache.append(), not by paged addressing itself.

• Logical page ownership is now tracked CPU-side to avoid those synchronizations.

• Paged attention has measured approximately:

  • Materialized: 55.1 tok/s
  • Paged: 57.5 tok/s
    on a relevant smaller models

Devlog-idk atp

Well i said that i would stop working on this project last devlog, butttt, i just saw this https://github.com/random-unknown-username/Thintensor/issues/1

Someone tried thintensor on a 128gb unified ram DGX spark, ANDDDDD now i wanna make it good enough like so its a finished thing

the website is more complete now

DevLog-3

Placement done, few things-

• back side has a few caps and stuff, because the space is tight
• the board is 22x22 rn, moving to a 25x25 soon, to make the board’s back completely empty and keeping all caps and stuff at the top!

Devlog-2

Completed the SoC design, now on to placement, a few things-

• The decoupling is just perfect to make the IC run and not burn
• Minimal stuff is used because of area constrains i plan to have
• Some GPIOs wont be exposed to SoM pins

Devlog-1:

Making the canva for video games!!!

The Tech Stack

I wanted this engine to run natively in the browser without massive downloads, so I chose:

• React + Vite for the frontend UI and state management.

• Three.js & React-Three-Fiber (R3F) for the 3D rendering pipeline.

• Cannon-es (@react-three/cannon) for rigid-body physics.

• Zustand for lightweight, blazing-fast state management.

• React Flow (@xyflow/react) for the Visual Scripting node editor.

  1. Building the ECS (Entity-Component System) in Zustand

Most React apps map data to UI components linearly, but game engines need an Entity-Component System (ECS) to handle hundreds of objects dynamically. I built a custom ECS using Zustand.

    // The core engine state powering the ECS
    interface EngineState {
      entities: Record<string, Entity>;
      addEntity: (name: string) => string;
      addComponent: (entityId: string, componentType: ComponentType) => void;
      updateComponent: (entityId: string, componentType: ComponentType, data: any) => void;
    }

This allows the engine’s Outliner and Inspector panels to dynamically edit any object in the scene in real-time, just like Unity or Unreal.

2. The Visual Scripting Compiler

The coolest feature I got working today was the Visual Script Editor. I used React Flow to let users connect nodes (like “On Update” -> “Move To Player”). But how does a node graph become executable game?

I wrote a compiler that traverses the node graph and translates it into a raw JavaScript string. This string is saved into the entity’s script component and executed dynamically in the game loop using a
Function constructor!

    // A snippet of the compiler output injected into the game loop
    } else if (node.data.label === 'Enemy: Move To Player') {
      output += `
        if (api.velocity) {
          const toPlayer = camera.position.clone().sub(mesh.position);
          toPlayer.y = 0;
          toPlayer.normalize();
          api.velocity.set(toPlayer.x * 4, -1, toPlayer.z * 4);
        }
      `;
    }

3. Bypassing React for Performance

The biggest hurdle today was the physics loop. Initially, the enemy tracking script updated the Zustand state every frame. With 15 enemies running at 60fps, it was triggering 900 React state updates a second, completely freezing the browser!

The novel solution for this was bypassing React’s render lifecycle entirely for physics. Instead of updating the state.entities transform data, the compiled scripts now directly tap into the raw Cannon-es physics
API (api.velocity.set()), mutating the physics hull natively. The performance skyrocketed from a frozen screen to a buttery smooth 60fps.

Devlog-1

Made the Power circuit with 3 JW5211 ICs each do-

• 5v -> 3.3v
• 5v -> 2.5v (closer to 2.6v for ddr1)
• 5v -> 1.1v

Required for the F1C100s from allwinner!
a small linux capable SoC

Also added a SPI flash and some status LEDs and stuff

Final devlog:

I would be moving from this project to other cool stuff i have in mind, the final benchmarks i got and everything, is pushed, had to do a lot of improvements to get bigger models running just as good as small ones

I know there can be a lot of other optmizations, but maybe a few months later ill come back to this, but for now, its gonna stay as is.

Devlog

this was originally supposed to be shipped to clubsite ysws but that is unfortunately no longer active, so I am just adding it here

A few cool things in the website-

• Playable games that were made by members (the raycaster demo was made by me for hackclub sprig!!!)
• this site would be under constant change for the next few days, we are finalizing a design with out school administration
• All animations were coded by me :D
• The logo and all art and stuff was design by other co-leaders from the club!

I figured I’ll just add the time i spent on stardance because the older clubsite ysws is inactive now

This devlog retains the apples-to-apples GPT-OSS control plus fresh Qwen3.5-9B, Llama 3.2 11B text-path, OpenReasoning-Nemotron-14B, and Phi-4

A LOT of optimizations were done to achive this results and 100s of benchmarks in different profiles and stuff, to get to this final table

ThinRuntime GPT-OSS-20B now achieves:

• 50.3445 tok/s median across three explicit 200-token runs.
• llama.cpp b9964 Q4_K_M: 40.3631 tok/s mean.
• ThinRuntime is 24.73% faster.
• Exact embedding and LM head; Q1/Q2 applies only to experts.
• Zero expert transfers during decode.
• Top-1 unchanged through step 4.
• Cosine: 0.987376 step 1, 0.983299 step 4.
• Accepted one-step configuration: 0.991415 cosine.
• llama.cpp Q4_K_M source-relative cosine: 0.792350.

Worked over a LOT of stuff, like doing apples to apples comparisions of models such as LLama 11b vision instruct, So over time I fixed a lot of stuff

added a new profile, that quants and quants and quants all layers till the model fits your VRAM, it doesnt care if the model is 20gb in weigths and you have a 8gb vram, it will quant and make it fit

So testing this autofit profile on LLama 11b vision instruct, heres what i found

it hit a cosine of 0.939758 and around 24tk/s on a 200 token test

Similarly i tested ollama Llama 11b vision instruct with q4_k_m quant, it got a cosine close to 0.720495 i would say it would be more closer to 0.8 because this cosine was for only top 20 logprobes that ollama exposes

This was done on
8gb vram rtx 5050 laptop
32gb ddr 5
ryzen 7 250
arch linux

and the comparision is apples to apples because the thinruntime also quanted the model aggressively