Made NGram class that can count occurences of the specified n grams in the corpus. Also, the tokenizer now has another module to map unique tokens to ints for memory efficiency. Now, I shall move onto estimating probabilities of each token given a context window.
Comments 0
No comments yet. Be the first!
Sign in to join the conversation.