What I made?

My model is trained. Final val loss is 3.28 after 3.5 hour, in comparison original GPT-2 was around 3.1 after 169 hours on 32 Google TPUs. Also my costed 18.8$ while OpenAi one 43.1k$. Technical info about training in previous devlog

What’s new in this devlog?

I added CPU support and CLI for inference. Most of new CPUs should works with it. It only works on windows.
Ready to ship!

CGPT

CGPT is a C++ and CUDA project for training transformer language models from scratch.

Training result

A 104M-parameter model was trained for one epoch on approximately 10 GB of FineWeb-Edu data, containing about 2.14 billion tokens. Training completed successfully after 58,874 optimizer steps with:

• final training loss: 3.302037;
• final validation loss: 3.282484;
• training time: approximately 3 hours 32 minutes on single Nvidia H100 GPU;
• estimated training cost: $25 .

The validation loss decreased from 3.908841 to 3.282484, showing clear learning progress without obvious signs of overfitting.

NOTE this is text prediction model not chatbot.

Performance

The custom BPE tokenizer processed the complete dataset in 22.62 seconds at approximately 414.98 MiB/s .After training, the model generated text at approximately 150–157 tokens/s on an NVIDIA RTX 3050 Laptop GPU with 4 GB VRAM. The generated text is generally fluent, but it can be repetitive and semantically unstable. This is expected for a relatively small model trained for only one epoch.

Output

The final model, checkpoints, tokenizer, and training logs are stored in:- output/model-104m;- output/model-104m/checkpoints/;- data/fineweb_edu_tokenizer_32k.json;- training_output.txt.

Generated output

The cat’s ring in front of the cage is short, so full-body cat with a small rackwork to keep them from licking tails from the cat’s face.
Close the cage, but not impossible.
Pine fronds are so thick that they can’t hold them up. They also don’t work as well as catnip.
But they are troublesome.
They tend to grow old and dead in the spring.
Pine fronds are the most beautiful cat in the West. They started life a few years ago at the museum, where they were sold in the strongest boxes in the

Provided images

On first image you can see val loss chart across steps, on the second image you have comparison with GPT-2 from 2019

Next steps in project

Now I will write CPU optimized kernels because now it only works on NVIDIA GPU.

Feature plans If I would have some money I would like to train something like 1.5B model on much larger amoount of data and time. I also would like to add post training focused on converastions so I would be able to chat with it.

Full report

For the complete training configuration, detailed results, generation samples, and GPT-2 comparison, see the full CGPT README.

I was working on optimizing attention and cross entropy kernel. I went from 41ms->1ms in attention backward
2ms-> 0.48ms attention forward and 3.5ms->2.3ms in cross entropy. On the first image you can see new kernels times on the second the old ones.

Note gpu utilization is not real in attention kernels

I was trying to train my model, but something went wrong. I need to find out, why loss jumped into sky.
Ln(32000) = 10.37 which mean tokens are random but I got 20.8 so model is predicting worst possible token???

Optimizations

I was working on optimizing CUDA kernels and adding some Doxygen documentation.

In the image you can see GPU utilization per kernel.

Note: The relatively low utilization is expected — these kernels are likely optimized for larger GPUs than the RTX 3050 (which is quite limited in terms of Tensor cores and SM count).

My small language model is written entirely in CUDA and C++ with no third-party dependencies.

Note I posted kernels bechamrk in the image because stardance markdown doesnt support tables yet

Successfully trained very small model. Optimized kernel a lot (especially attention backward). Now I have to optimize kernels even more, documented code, and train full model.

Added embedding backward, attention backward, linear backward, rmsnorm backward, rope backward, swiglu backward

Hello, world!

Recently, I’ve been working on optimizing CUDA kernels for my own LLM implementation. Most of them are now performing quite well, especially considering that I’m running everything on an RTX 3050 Laptop GPU with only 4 GB of VRAM.

There is still one kernel that keeps frustrating me: GQA attention. Its GPU utilization is much lower than the rest, and I’m not sure how to optimize it further.

Here are my current benchmark results:

RMSNorm

• Input: [512, 4096]
• Data type: FP16
• Execution time: 0.049 ms
• GPU throughput: 89%

SwiGLU

• Input: [512, 4096]
• Data type: FP16
• Execution time: 0.074 ms
• GPU throughput: 90%

Linear layer

• Input: [512, 4096]
• Weights: [4096, 4096]
• Output: [512, 4096]
• Data type: FP16
• Execution time: 1.04 ms
• GPU throughput: 60%

Note: The reported throughput is relatively low because this benchmark runs GEMM on a comparatively small input matrix. At this problem size, the GPU cannot reach the same utilization as it would with larger batches or longer sequences.

Rotary Position Embedding — RoPE

• Query: [1, 512, 32, 128]
• Key: [1, 512, 8, 128]
• Cosine and sine cache: [512, 64]
• Data type: FP16
• Execution time: 0.06 ms
• GPU throughput: 87%

Grouped Query Attention — GQA

• Query and output: [1, 512, 32, 128]
• Key and value: [1, 512, 8, 128]
• Data type: FP16
• Execution time: 3.32 ms
• GPU throughput: 10%

This is currently the main bottleneck. I have already applied tiling, online softmax, and shared-memory optimizations, but the utilization is still very low.

Embedding lookup

• Number of input tokens: 512
• Vocabulary size: 32,000
• Hidden size: 4,096
• Output: [512, 4096]
• Data type: FP16
• Execution time: 0.05 ms
• GPU throughput: 88%

Summary

Overall, I’m satisfied with the results, but the attention kernel still needs a lot of work. (I will probably use cuDNN). The hardest thing before me - backprop. But now I can play with kernels and model layers.

Implemented RoPE kernel. All 10 tests passed. I also added in code documentation. Netx big stepp will be attention kernel.

With chatgpt help i optimized my tokenizer. 100MiB needs only 33.72s of training time. Batch encode throughput is around ~400MiB/s which is around 11.5x faster than OpenAI tiktoken(look at images). I have CPU AMD 5 5600H inside PC. I think that on AI data center my tokenizer could reach even about 4GiB/s ! My tokenizer is fast but it require relative big amount of ram. Nedded RAM is about 20x of train file size. When I finish my project I will train my model and benchmark my tokenizer on god tier machine. Wish me luck!

Succesfully improved Tokenizer. Now 100MiB of text encode in around 1 minute. In previous devlog 8MiB nedded 22 seconds. So huge improvemnt

Optimized tokenizer - 100Mib require 21.8s training time. Encoding time 5.5MiB/s

Implemented RMSNorm, linear layer kernel. Implemented BPE Tokenizer - currently its slow.

Successfully complied CUDA hello world