CGPT

CGPT is a C++ and CUDA project for training transformer language models from scratch.

Training result

A 104M-parameter model was trained for one epoch on approximately 10 GB of FineWeb-Edu data, containing about 2.14 billion tokens. Training completed successfully after 58,874 optimizer steps with:

• final training loss: 3.302037;
• final validation loss: 3.282484;
• training time: approximately 3 hours 32 minutes on single Nvidia H100 GPU;
• estimated training cost: $25 .

The validation loss decreased from 3.908841 to 3.282484, showing clear learning progress without obvious signs of overfitting.

NOTE this is text prediction model not chatbot.

Performance

The custom BPE tokenizer processed the complete dataset in 22.62 seconds at approximately 414.98 MiB/s .After training, the model generated text at approximately 150–157 tokens/s on an NVIDIA RTX 3050 Laptop GPU with 4 GB VRAM. The generated text is generally fluent, but it can be repetitive and semantically unstable. This is expected for a relatively small model trained for only one epoch.

Output

The final model, checkpoints, tokenizer, and training logs are stored in:- output/model-104m;- output/model-104m/checkpoints/;- data/fineweb_edu_tokenizer_32k.json;- training_output.txt.

Generated output

The cat’s ring in front of the cage is short, so full-body cat with a small rackwork to keep them from licking tails from the cat’s face.
Close the cage, but not impossible.
Pine fronds are so thick that they can’t hold them up. They also don’t work as well as catnip.
But they are troublesome.
They tend to grow old and dead in the spring.
Pine fronds are the most beautiful cat in the West. They started life a few years ago at the museum, where they were sold in the strongest boxes in the

Provided images

On first image you can see val loss chart across steps, on the second image you have comparison with GPT-2 from 2019

Next steps in project

Now I will write CPU optimized kernels because now it only works on NVIDIA GPU.

Feature plans If I would have some money I would like to train something like 1.5B model on much larger amoount of data and time. I also would like to add post training focused on converastions so I would be able to chat with it.

Full report

For the complete training configuration, detailed results, generation samples, and GPT-2 comparison, see the full CGPT README.