LLM
- 3 Devlogs
- 18 Total hours
Minimal SLM for English
@phattar4phan
on
LLM
2h 43m 47s logged
Modify training configurations, pipeline and fix some error in src/model/mlp.py and src/model/rope.py
@phattar4phan
on
LLM
4h 19m 29s logged
Change the dataset from entirely scratch to Dolci-Instruct-SFT, then also add serializer (src/utils/serialize.py) to serialize the dataset into .txt file and then to train it with spm.SentencepieceTrain() (src/utils/tspm.py) also add bin.py to turn tokenized into token IDs for faster dataloder, also removed unnecessary file
@phattar4phan
on
LLM
10h 44m 9s logged
#1 - Been on configurating the project, also after configuration I go on getting the dataset, while you can search for them online, I wanna be really type of ‘from scratch’ so in this case I also create the dataset from scratch as well. And all those 10hrs 44m, I’ve been manually entering random wikipedia articles to the code to scrap the texts, The main obstacle that make this so long is that sometimes I have typos, wrong words, or even not found which I’ve to also manually resolve all of them. Thats all but been going on around ~1600 articles
