Trained the wavenet architecture on a file with just shakespear in it. It’s got the structure down well, and sometimes can do words and wordlike outputs, but generally spouts straight gibberish. I will try to follow a lecture to build a better Tokenizer and Transformer next time. Right now each character is treated as a token, and embedded into 10 dimensions, it would be better if pairs of characters were treated as tokens and embedded. Increasing the context size could also help, it’s currently an input of the previous 8 characters and outputting 1. This is it’s output: