Fixed up the readme a bit in preparation to ship. Specifically, I added info on what all the files do, but I named them poorly and then forgot what most of them do so it’s kind of confusing.
I think you are just lacking data. I’m currently trying the same project and my model managed to produced grammatically correct text after I fed a portion of openWebText.
Also you might want to check out Kaggle or Google Collab for training part since running training locally is abysmally slow.
Perhaps it’s a bug in your code ? Have you tried overfitting a small text sample ? Like one text of ~ 1 M tokens ?
If even after that and increasing the learning, your model still don’t learn, you should definitely check your code.
Comments 5
I think you are just lacking data. I’m currently trying the same project and my model managed to produced grammatically correct text after I fed a portion of openWebText.
Also you might want to check out Kaggle or Google Collab for training part since running training locally is abysmally slow.
Be wary that KAGGLE IS EXTREMELY DIFFICULT to work with when you’re a beginner
Side note: I really admire your dedication. Good job
@Definitly-Not-Me Thanks for the tips, I might try that out. I have a ton of data, so I doubt that’s the problem, but I might try adding more.
Perhaps it’s a bug in your code ? Have you tried overfitting a small text sample ? Like one text of ~ 1 M tokens ?
If even after that and increasing the learning, your model still don’t learn, you should definitely check your code.
Sign in to join the conversation.