By
@kazenoko
Meant to be a Diffusion-based Code Completion model. Somehow turned into research on scaling Masked Diffusion Language Models instead. I built, trained, optimized, broke, retrained, and tested multiple models while investigating how training-token requirements change as MDLMs get larger.
This was supposed to be a weekend project to learn LLMs. It became independent research on scaling Masked Diffusion Language Models instead. I built the models, survived datatype/timestep nightmares, optimized MLX on Apple Silicon, achieved ~1.5–2× faster local training, and investigated how training tokens scale with model parameters.
To test the models or look at the research, head on over to https://telos.research.wingit.tech/
(please don’t expect the models to be good - they are NOT. They’re small models that barely generate anything useful [clearly, shown by the screenshot] and the only useful thing about them is how much the correct answer probabilities improve with increase in parameters and tokens, do note that inference is also pretty slow.)
This was meant to be a weekend project for me to learn basics of LLMs with…
Instead I’ve managed to work on Diffusion Language Models and perform research on it considering there’s not much (or enough) knowledge on Diffusion Language Models to begin with (as far as I’ve seen).
So here’s what I’ve done instead of being normal:
An absolute pain of a research project but maybe worth it.
This is the testing page and research paper.
Had to deal with nightmares like timestep distribution, datatype mismatch and running out of compute all the time and optimising for MLX, but guess I managed to (somewhat) answer the primary question I had for Masked Diffusion Language Models:
The full answer, methodology, and evidence are in the report linked above.
But if you’re lazy and just want the current empirical relationships, here they are (these are still preliminary and may change as the research continues):
Where R(N) is the optimal token to parameter ratio for a model of N parameters (in millions)
Somehow this was a weekend project. That ended up taking more than a week and probably take even more time than this for a full report.
