Devlog 0: Build a Large/Small/Whatever LM

Overview

Hello Stardance! Hello world appetIntern disdainwartsnine.
Those were the first words of my Language Model, am feeling so fogging proud right now. I’m trying to get into ML for the next year (not just because of the hype). For my first projects, I tried building a simple perceptron as well as understanding the algorithm behind Tic-Tac-Toe. Nothing grandiose. Why aim so high for my next project since I’m not that knowledgeable ? Because I’m short of time and there are full tutorial available, that’s all.

Goal & MVP

The MVP goal is to be able to generate coherent text (will probably be gibberish since I’m not a billion dollars corpo) and implement simple RAG to compensate for the lack of intelligence.

I plan to upgrade it as I learn more about LLMs and Transformers.

The Current State

I just got the basic unoptimized architecture down.
It tooks so much time to understand the Transformer architecture and all the maths that goes behind it (thanks to 3Blue1Brown it was manageable). Now I need to train & evaluate the “Advanced Autocomplete Machine” so that It can start acting more like GPT2.

Picture:

Liking my rice ? 😏