I finished training it! It finished with a val loss of around 1.76, it got down to 1.73 but ticked up at the end. The output is pretty good, but not substantially better than the wavenet model. However, it’s vaguely word-like, and took 20 hours to train so I’m not sure how much larger I can make the model before my laptop implodes. Next I will probably try to find a question-answer training dataset, so that I can start training it to answer questions. This step that I’ve been working on is the pre-training step, where I teach the model to basically spit out documents. Next, I have to have it spit out documents that happen to be question-answer documents.