After debugging for a week. It’s finally done. There’s a whole story behind this though.

1. Bundling a python project with pyinstaller

To ship my project I had to create an exe file out of my project, and for that I used pyinstaller. Ran into multiple micro-dependencies issues like groovy and safehttpx. And the exe file was over 6 GBs

2. Pytorch CPU hell

To reduce the exe file size, I had to use the cpu only torch packages. But when pyinstaller bundled those, I ran into some dependency issue by c10.dll file. And no matter how many workarounds and things I tried I just couldn’t fix it. I just knew at that moment that this is just not the way. There were also some issues regarding Intel OpenMP thread collision, I don’t even know what that means! :(

3. ONNX the savior

Researching a bit I stumbled upon onnx runtime engine for pytorch models. So, I created another app script this time with onnx model loading and numpy based preprocessing (sadly I couldn’t even use the preprocessing pipeline from pytorch). But this was simply the beginning….

4. ONNX dependency hell and incompatibility with other packages

When I tried building the onnx based app, the first issue was a silent failure, no log, no warning just exe file not running at all… It was some silent C level crash that python couldn’t detect, so I changed my app script again and added fault_handler. There was a raw memory collision between gradio and onnx for dll space. I changed the import order for onnx first but pyinstaller still hooked up onnx runtime dll’s to some third party package’s dll files.

Finally after that, I manually wrote a hook script and grabbed the dll files manually myself to inject into pyinstaller analysis object.

Devlog 1.5

• Wrote readme
• Had to commit a couple of times cuz I kept misspelling/misformatting my file names xD.

Devlog 1.4

• Added app usage description
• Added examples for quick testing
• Chose a better theme

Devlog 1.3

• Made a gradio based wrapper around the model
• Debugged the app deployment on hugging face
• Handled edge cases and refactored the code

Devlog 1.2

I mainly focused on profiling the model’s performance, and I found the model’s inference to be taking ~200 ms on GTX 1650. But I compiled the model with torch.compile() which basically fuses all the cuda operations in one big monolithic kernel and my jaw dropped seeing 20x performance boost taking only ~8 ms.

Oh and I also profiled the compiled model wrong the first time and also recorded the lazy jit compilation of the model during the first pass. The solution of it was to synchronise cuda with CPU and then run the profiler.

(This took wayyy more than just 31 mins, I had to install and set up WSL to run torch.compile because pytorch’s backend uses triton which doesn’t support windows).

DevLog 1.1

• Fixed Bugs
• Wrote a training ipynb file so I can train the model on google colab or kaggle
• Tried different hyperparameters for better generalization and training efficiency
• Edited .gitignore

First off, I needed to understand how ResNet worked so I read the paper: https://arxiv.org/abs/1512.03385

After reading made a plan and visual to follow and implement in the code, starting off simple I choose to implement ResNet-18.

In code:

• Downloaded Cifar-10 (this took a long time somewhy)
• Defined the Residual Blocks
• Encapsulated the Resnet Model
• Wrote training script and some utility functions
• Tested inference and shape match

Changes

• Made the I/O interface more colourful
• Pushed the binary to repo, and attached it with the release
• Some bug fixes

Final touches

• Added a bigger corpus i.e. wikitext2 by salesforce AI Research.
• Credited corpus and linked associated license.
• Wrote a python script to read the parquet format corpus and turn it into .txt format.
• Some minor changes in code.

Progress

• Better CLI interface
• Wrote Readme for shipping.
• Bug fixes (unigram model input vectors processing without checks, etc).

Implemented Model Evaluation with Perplexity

Perplexity (a metric borrowed from information theory) acts a summarized evaluation metric for your model, that intuitively evaluates how much your model branches off in generating texts.

Mathematically, it’s a geometric mean of the inverse probabilities as calculated by the model, normalized by sequence length.

For seen texts, my model achieved a perplexity around 20-30, and for unseen texts it varied alot due to random picking of tokens in that case.

Implemented Laplace Smoothing in the model

Smoothing refers to generalization of n gram model to texts they haven’t seen before. Laplace smoothing (also known as add-one smoothing) is the simplest type of smoothing that avoids the probability estimates to reach zero when handling unseen token or sequences.

It is quite easy to implement this in code, just add one in the numerator of probability formula and add vocabulary size in the denominator.

• Implemented probability estimates of each word using the formula:
  Counts of gram sequence in corpus/ Counts of context window sequence in corpus

• Improved the I/O logic a bit aswell, the debug statements are there but commented.

• Made module for estimating probabilities of words given a context window using MLE (Maximum Likelihood Estimation).
• Fixed the mapping of tokens back to strings, by using temporary dynamically allocated array and copying it into vector.

Made NGram class that can count occurences of the specified n grams in the corpus. Also, the tokenizer now has another module to map unique tokens to ints for memory efficiency. Now, I shall move onto estimating probabilities of each token given a context window.

Overview: N Gram Model is a fundamental model in Natural Language Processing used to predict next word in a given sequence. It relies on markov’s assumption that the probability estimate of a word relies on it’s immediate preceding words rather than the whole text/sentence.

What I have made uptill now: I have made the first module required for this project which is a Tokenizer, a tokenizer takes a text corpus and separately stores each word or token in a lookup table.

Future Direction: The core mechanism of N gram models, which is probability estimates for each n word sequence using frequency found in the corpus. Further, I would want to add smoothing to these probability estimates and add model evaluating metrics like perplexity.