My own AI

Devlog #7
Hello guys! This is probably my last devlog before releasing this project, and I want to share what happened during the last 3 days.
Also, keep in mind that I previously had a version based on a Q&A-type database that I had generated using an LLM.
First off, the version from the last devlog was based on a database from DailyDialog. It was very heavy for the model’s “brain”, and as a result, it wasn’t really able to communicate properly. It would sometimes respond with something completely unrelated to the question.
After that, I tried training it on a Wikipedia dump. As an idea, it was good, but the amount of time it took to finish a training session with 30 epochs was LONG. And even after that, the AI still wasn’t capable of having proper conversations.
After that, I asked ChatGPT for help, and I can say that ChatGPT is not only bad, it is horrible for debugging. It makes stuff up, changes things that aren’t even broken, and generally makes the debugging process harder.
So I tried using Claude. Claude at least tried. It actually computed the project on its server, fixed something, but it still wasn’t enough.
Then I discovered Codex. It’s not entirely new, but I didn’t know about it before!
I gave Codex the project files and asked it to search for the problem. It gave me the files back, and it actually fixed the problems.
When I tried the fixed version, I asked it some questions. It had been trained on both a Q&A database and a Wikipedia database, and it actually answered correctly most of the time.
The problem was that when I asked something that wasn’t in the Q&A database, it retrieved information from the Wikipedia dump. That wasn’t necessarily bad, but it didn’t know how to actually make a sentence using the information. It was basically just a retrieval tool rather than a conversational AI.
Since my Codex tokens expired and I can’t use it for a month, I didn’t bother continuing with that version.
I went back to the version from before the last devlog, which was based on a smaller Q&A database. I improved the database, retrained the model, and now it actually responds.
The problem is that if you’re not using proper grammar, it can still respond with something unrelated to the topic. And if the topic isn’t included in the database, it can make stuff up.
After that, I added an Ollama version alongside my trained model. This way, if the user wants an answer that my trained AI can’t provide, they can use the Ollama model instead.
I also made a UI with the help of an LLM.
So now the app is functional. Both models are available.
I also thought about adding a tool to my trained AI that detects when the user wants math support, but for now it would only be for math and not other topics, so I’ll see if I end up adding it.
About web support
I don’t think I’ll add a web version since the UI is made using PySide6, and making it web-compatible would basically mean rewriting the whole UI.
Or maybe there’s an easier way to do it that I don’t know about.
Linux and macOS support
Linux and macOS support is possible, but I don’t have a Mac to actually test whether the program works properly on that platform.
Linux should be easier since I can install a Linux distro in a VM and test it there, although I’m not completely sure yet.
And that’s basically where the project is right now.
If you have any suggestions leave them in the comments, thanks!

Devlog #6
Hello guys!
Yesterday I realized what was the problem with the language model and its predicting. So basically I was testing it by typing “Hello” without punctuation and so the AI was confused and did not know how to respond, hence the unexpected responses. When I tried with the word “Hello.” with the dot it did work! And besides this problem, I also encountered another problem. The AI couldn’t recognize, even if the word was in the training.txt file, if it was spelled wrong or had capital letters or not where it should have been. For example I asked it “What is EArth?” and it returned nothing because it did not recognize “EArth”, in its vocabulary, so last night I sampled a training text in json format from DailyDialog dataset, made a converter.py file, which extracted the dialog from train.json . I ran 3 epochs because the new text contained 2660291 training examples (WHICH was WAY WAY bigger than I expected!), and it took like 1,5 hours to finish. After that I tested it and again, the model predicted the answer wrong so now I upgraded the embedding size from 32 to 64 and ran again train.py (currently it is still training!) . Anyway, if it doesn’t work again I will probably look into the configuration again, maybe upgrade the epochs from 3 to 6, since the other major components like main.py model.py and train.py are fine (probably). I attached below the training progress

Devlog #5
My brain is soup currently. I have made real progress, at least, at least… . Now the AI is trained on my CUDA cores from my GPU. And now it can actually make a conversation, which is WOW, but the thing is… the responses are not consistent or they do not make sense, but at least they are words that can be understood separately. I have also changed train.py, model.py and main.py multiple times, upgraded to training.txt file which now contains multiple conversation examples, but the project is far from over. I attached below how it responds currently. If it weren’t for the 10 hour devlog limit I would’ve kept going.

Devlog #4
Hello guys!
• I finally rebuilt the 3 biggest components of my AI, those being main.py, model.py and train.py, completing a solid foundation which can be upgraded from now on without the need of deleting the current code completely.
• Anyway, currently the text sample which it is being trained on is not too big, so it can’t “learn” much, and the context right now is just “Hello wo”, and it predicts the next character right as intended, using the last 8 characters
• The first changes I obviously need to make are changing the context from that to the input of the user and making the training sample bigger, then we can move to upgrade the epoch number from 100 to lets say more and the context size from 8 to more characters. I will announce my plans for the future upgrades in the upcoming devlogs.
• Here are some previews of the code and some notes I took: (if they all load correctly hopefully)

Devlog #3
Hello guys!
I am working so the program will be based on an 8 character architecture, meaning that it will now predict the next character using the last 8 characters, not just the last one, meaning its brain gets upgraded! I am still learning how to do an AI at home with the help of YouTube tutorials and AI so it will take many hours to finish, but my expectations for the final product are high!

Devlog #2
Hello guys!
I have updated main.py which tests the AI, train.py, model.py and tokenizer.py and I reached a milestone since the AI now predicts what letter comes next. Right now it is being trained by a pretty simple text file but I’m planning to upgrade that too!
Here’s a quick preview:

Devlog #1
Hello guys today I am working on a project with the goal with building my own AI, yes, you heard that right! The goal is to train it locally and that it works 100% offline
Here’s what I did up until now:

1. I set up the project in python
2. Built my first neural network with the help of AI that contains one neuron and which was used to understand weights, loss, prediction and more
3. Did my first AI training which the AI discovered the multiplying by 2 rule
4. Dataset loader
5. Tokenizer
6. Custom vocabulary
7. Encoder/Decoder and a few other stuff