This was meant to be a weekend project for me to learn basics of LLMs with…
Instead I’ve managed to work on Diffusion Language Models and perform research on it considering there’s not much (or enough) knowledge on Diffusion Language Models to begin with (as far as I’ve seen).

So here’s what I’ve done instead of being normal:

Research on Scaling Masked Diffusion Language Models

An absolute pain of a research project but maybe worth it.

This is the testing page and research paper.

Had to deal with nightmares like timestep distribution, datatype mismatch and running out of compute all the time and optimising for MLX, but guess I managed to (somewhat) answer the primary question I had for Masked Diffusion Language Models:

How do tokens scale with parameters for MDLMs?

The full answer, methodology, and evidence are in the report linked above.
But if you’re lazy and just want the current empirical relationships, here they are (these are still preliminary and may change as the research continues):

1. R(N)=14.43⋅ln(N)−21.44
2. R(N)=6.83⋅√N​−9.14
3. R(N)=2.39⋅N^(0.74)

Where R(N) is the optimal token to parameter ratio for a model of N parameters (in millions)

Somehow this was a weekend project. That ended up taking more than a week and probably take even more time than this for a full report.