tinyapper
- 11 Devlogs
- 32 Total hours
LLM that is small, simple, and can be tinkered with!
Ship #2
Devlog 11: Cleaned this up a bit more, adding session attached KV caching and more polish for one last ship. School + apps is getting busy, so I think this will be my final devlog. TYSM stardancers, I had lots of fun creating tinyapper 
Devlog 10: Created the website for my chatbot! Other than retraining, I think the last steps for me look like implementing a session attached KV cache. 2.0 soon :D
Devlog 9: An epoch of ultrachat finished faster than I thought it would… tinyapper really likes chocolate chip cocktails I guess 😭. Looping and other problems are getting really prevalent, I think I’m hitting the limit for a 50M model.
Pre devlog thank you: You guys really did fund my cloud credit,
17.57x multiplier for my first ship on this is crazy! I read through the
feedback, my plan now is to use Stardust for cloud credit (pending
order) to properly pretrain the model and finetune it using Ultrachat (I
hear yall :3). Thanks for all the support and suggestions here and on
Slack, it really motivates me to keep coding <3
Ship 1.0.1 (Devlog 8):
- Implemented variable learning rate: in theory lets the model hit
slightly lower numbers and train longer - Added a validation split and test: long overdue lol
- Added data processing and support for Ultrachat: allows me to
circumvent processing data on the cloud - Added L4 settings for cloud: yes I’m getting ready to train on the
cloud - Updating docker container to not include a model + other random stuff: volume mount on
runtime to prevent pulling 2gb from ghcr every update - Retrained model: WE BEAT OPENAI RAHHHH (from 7 years ago), we are now
at 3.09 nats (for validation too, which is a better metric)! For
reference, GPT 2 small gets a loss of 3.4 nats when evaluated without
extra training on the same dataset (we don’t talk about the fine tuning
results)
Ship #1
Devlog 7: I optimized streaming! Not only that, my mini PC has integrated graphics that apparently makes LLM go zoom, so I got a really large unanticipated speed boost :0. I got to increase the generated tokens all the way to near max, so y’all can get more gibberish faster! Going to ship this version :)
Devlog 6: After all this time… my frontend is the exact same! It’s a lot faster though! I spent a lot of time trying to optimize my model, from KV Caches to RoPE and quantization. Overall, KV caching and streaming text seems to give the best experience, so that’s what I went with. Seeing if I can deploy to nest or further optimize streaming…
Devlog 5: Finally implemented a cache… now I just have to rewrite my attention module to work around it…
Devlog 4: I tried to get away with not using a KV Cache and ship today… but CPU said no. Considering using cloud to host this until I implement a KV Cache, but I know I should just code one up. Why do LLMs need so much compute 
Devlog #3: The model is trained! I will fine tune and train it over next week, but I think I want to get a first ship and demo first. Yapperpedia coming soon!
Devlog 2: Tinyapper (an llm from scratch) is done with its first train! After an epoch of training it averaged ~3.8 nats/token for training data, which is decently close to GPT 2s 3.62… I hope to turn this into a frivolous Wikipedia article generator this weekend for a first ship this weekend, so get ready for that!
Devlog #1: Nowadays, LLMs are such a popular technology with a lot of theory, but very few people seem to have implemented one. I wanted to create my own LLM to tinker with on my own, and so far cross entropy seems to be approaching human language! Wish me luck <3
