4h 14m 8s logged
3h 11m logged
Two steps before the throne
Finally deployed the training script to Kaggle. Now all there’s to be is wait and hope.
A quick list of what’s new:
- Divided the training loop in many functions to handle dataloading, parsing, logging and more;
- Managed to find a copy of gutenberg books to serve as dataset;
- Implemented checkpoints in the training so that I will not lose all progress if I run out of compute;
- Dataloading is now done using
np.memmapensuring that only a portion of tokens are loaded in memory; - Scaled the value of hyprparameters based on configuration of gpt-2 small (my main source is Chatgpt/GLM and a rough guess of number of epochs);
- Fixed a “storage leak” where the best model checkpoints where saved per epoch instead of overwriting previous one (totally my fault)
Next:
- Fix my nonexistant git history while waiting for the training to finish
7h 45m 53s logged
Small update on building a Language Model
So after getting the architecture down and adding collecting some data, I started the training phase. That was one week ago ….
Training is surprisingly (or not ?) actually more than just shoving in more data and expect it to work. Right now I’m struggling with finding sensible hyperparameters and getting the model to output something coherent.
What’s new
I mainly worked on the training loop and minor optimizations to hidden layers to distract myself while Pytorch was frying my APU. Of course there were also bugs in the code that had to fix on the way.
For monitoring the training I decided to use plain TUI built with Rich but I’m currently adding a csv logging for better history tracking.
At first I was planning to run the training on my laptop but I understood why billions dollar companies needed so many datacenters. Anyway, I deployed it on Kaggle which also was harder than it should be.
Next
- Take a break,
- Improve training loop,
- Deploy on Kaggle
- Forget about it
7h 38m 40s logged
Devlog 0: Build a Large/Small/Whatever LM
Overview
Hello Stardance! Hello world appetIntern disdainwartsnine.
Those were the first words of my Language Model, am feeling so fogging proud right now. I’m trying to get into ML for the next year (not just because of the hype). For my first projects, I tried building a simple perceptron as well as understanding the algorithm behind Tic-Tac-Toe. Nothing grandiose. Why aim so high for my next project since I’m not that knowledgeable ? Because I’m short of time and there are full tutorial available, that’s all.
Goal & MVP
The MVP goal is to be able to generate coherent text (will probably be gibberish since I’m not a billion dollars corpo) and implement simple RAG to compensate for the lack of intelligence.
I plan to upgrade it as I learn more about LLMs and Transformers.
The Current State
I just got the basic unoptimized architecture down.
It tooks so much time to understand the Transformer architecture and all the maths that goes behind it (thanks to 3Blue1Brown it was manageable). Now I need to train & evaluate the “Advanced Autocomplete Machine” so that It can start acting more like GPT2.
Picture:
Liking my rice ? 😏
Ship
MiShell
I tried to implement a basic UNIX shell as an introductory project to C. It supports only piping and got some builtins. That’s all I could do without having to write a lexer/parser myself. Prior to that the most impressive thing I did with C was checking if a string is palindrome or not.
Challenges encountered:
Manual memory allocation:
As someone used from the simplicity of Python, it was quite the hassle but I did learn awesome strategies to manage memory namely arena and lifetime. I’m kinda proud my code doesn’t explicitly use malloc and that Valgrind confirmed it was leak-less. Even tho nobody will use it (and I know there are bugs), I can see that I progressed. I would love to explore more memory management techniques later.
Segfault:
Even with ton of flags I couldn’t escape those. I kinda convince some of them are still waiting for me, there, in one of my function call.
Manpages:
There are just so confusing. I understand better now why Stackoverflow is goated.
Git:
I didn’t even knew there was an hygiene for good commit. Know I think I know a little more? I will probably need to work on that too.
How to test my project:
Just download the latest release on Github. You should be able to run it on Linux and windows (via WSL). Mac I don’t know. But OS X is Unix-based so perhaps it will works too ? If it crash, …
let’s not talk about such sad things.
5h 55m 11s logged
Final Devlog:
My shell now started to look like something, although I know that it’s not even close to be prod ready. I did my best to hack together something usable i.e that compiles. If I were to add more than that and I would need to create a whole programming language. Even tho I would love to, that’ll be for another day…
Oh, concerning new features, I added the GNU readline library (same used by other shell) so keyboard navigation actually works like a real shell.
I also used another library to handle the parsing instead of my custom parser (it served me well) and finally I customized the prompt to make it less generic.
Now Mishell also support basic piping.
😮💨 Took me soooo long just because I didn’t google enough. It was a good learning experience.
Now, I can finally sleep …
…
…
….
I can sleep now right ?
“C is quirky, flawed, and an enormous success.”
- Dennis M. Ritchie.
7h 39m 3s logged
Diary of a masoschistic perfectionist C noob:
Hello everyone. Hope you’re enjoying the event. On my end things got a little be nasty for the past days.
Truthfully, the challenge was not that hard but you know how it goes:
man gotta invent lemons just because he can right :)
I fell down the C rabbit whole so bad: macros, pointers, errno, free, linking, … The hardest part was obviously managing memory. I was conditioned so much by Youtube to be scared of memory leaks that I had to rewrite an allocator just so to not use malloc. The other challenge I created for myself was to copy-paste random compile flags from everywhere to ensure my code would be mEmORy SaFe:
Spoiler: didn’t even work :(
I dearly missed Python simplicity. I will just have to fix my skill issues.
Anyway, next challenge will be to improve allocation (again) and add more features to the shell. I’ll probably be done (mostly) after this devlog since the purpose of the shell (interact with cli tools) is accomplished.
Now I can finally sleep.
If debugging is the process of removing software bugs, then programming must be the process of putting them in.
2h 26m 2s logged
MiShell is a very simple and dumb shell. It got some builtins commands
and … that’s all. I plan to improve it later (if I find time)
