N Gram Model
- 9 Devlogs
- 13 Total hours
A Basic Model for natural language processing in C++
Ship #1
Implemented Model Evaluation with Perplexity
Perplexity (a metric borrowed from information theory) acts a summarized evaluation metric for your model, that intuitively evaluates how much your model branches off in generating texts.
Mathematically, it’s a geometric mean of the inverse probabilities as calculated by the model, normalized by sequence length.
For seen texts, my model achieved a perplexity around 20-30, and for unseen texts it varied alot due to random picking of tokens in that case.
Implemented Laplace Smoothing in the model
Smoothing refers to generalization of n gram model to texts they haven’t seen before. Laplace smoothing (also known as add-one smoothing) is the simplest type of smoothing that avoids the probability estimates to reach zero when handling unseen token or sequences.
It is quite easy to implement this in code, just add one in the numerator of probability formula and add vocabulary size in the denominator.
-
Implemented probability estimates of each word using the formula:
Counts of gram sequence in corpus/ Counts of context window sequence in corpus -
Improved the I/O logic a bit aswell, the debug statements are there but commented.
- Made module for estimating probabilities of words given a context window using MLE (Maximum Likelihood Estimation).
- Fixed the mapping of tokens back to strings, by using temporary dynamically allocated array and copying it into vector.
Made NGram class that can count occurences of the specified n grams in the corpus. Also, the tokenizer now has another module to map unique tokens to ints for memory efficiency. Now, I shall move onto estimating probabilities of each token given a context window.
Overview: N Gram Model is a fundamental model in Natural Language Processing used to predict next word in a given sequence. It relies on markov’s assumption that the probability estimate of a word relies on it’s immediate preceding words rather than the whole text/sentence.
What I have made uptill now: I have made the first module required for this project which is a Tokenizer, a tokenizer takes a text corpus and separately stores each word or token in a lookup table.
Future Direction: The core mechanism of N gram models, which is probability estimates for each n word sequence using frequency found in the corpus. Further, I would want to add smoothing to these probability estimates and add model evaluating metrics like perplexity.
