tinyapper
By
@craisin
- 2 Devlogs
- 4 Total hours
LLM that is small, simple, and can be tinkered with!
Devlog 2: Tinyapper (an llm from scratch) is done with its first train! After an epoch of training it averaged ~3.8 nats/token for training data, which is decently close to GPT 2s 3.62… I hope to turn this into a frivolous Wikipedia article generator this weekend for a first ship this weekend, so get ready for that!
Devlog #1: Nowadays, LLMs are such a popular technology with a lot of theory, but very few people seem to have implemented one. I wanted to create my own LLM to tinker with on my own, and so far cross entropy seems to be approaching human language! Wish me luck <3
