Devlog #6
Hello guys!
Yesterday I realized what was the problem with the language model and its predicting. So basically I was testing it by typing “Hello” without punctuation and so the AI was confused and did not know how to respond, hence the unexpected responses. When I tried with the word “Hello.” with the dot it did work! And besides this problem, I also encountered another problem. The AI couldn’t recognize, even if the word was in the training.txt file, if it was spelled wrong or had capital letters or not where it should have been. For example I asked it “What is EArth?” and it returned nothing because it did not recognize “EArth”, in its vocabulary, so last night I sampled a training text in json format from DailyDialog dataset, made a converter.py file, which extracted the dialog from train.json . I ran 3 epochs because the new text contained 2660291 training examples (WHICH was WAY WAY bigger than I expected!), and it took like 1,5 hours to finish. After that I tested it and again, the model predicted the answer wrong so now I upgraded the embedding size from 32 to 64 and ran again train.py (currently it is still training!) . Anyway, if it doesn’t work again I will probably look into the configuration again, maybe upgrade the epochs from 3 to 6, since the other major components like main.py model.py and train.py are fine (probably). I attached below the training progress