well, today was a disaster. 8 ours that didnt really account for much to be honest :(

I have tried to move from qwen to minicpm 1b, a model that ALLEGEDLY is a lot better at using tools to gather data rather than using its internal baked in weights. Well, apparently thats not true. I have spent hours making the tools, adapting the outputs and everything for that model, just to then realized that “wow people are wrong and im so tired of ai bros hyping stuff up for nothing!”. So, in the end, i reverted to the qwen models, since they are decent enough for this. Even the 0.8b model gives better results than minicpm…

Well, at least thats not all i did. After reverting to a older commit, i have re implemented all the improvements that dont have to do with minicpm. You can now see the sources of where the llm fetched data from under the response bubble, and you can click them to open up the pages and double check the information and stuff. Also, i have worked on large file handling. Previously, for large files it would try to read the entire thing at once. That is insanely inefficient for large files, so i have added a new tool that lets it read from line x to line y. Like that it can only retrieve relevant stuff and not flood its context. I have ALSO worked on compatibility. Now you can change between cpu and gpu inference. If you have under 6gb of vram, the gpu option is grayed out. You might say “thats a little overkill”, but you have to take into account that context also eats up vram, and quite a lot as well. So now you can use the cpu, which uses your system ram, to reliably have long conversations. Sadly in the demo it ended up crashing the app and freezing the process with a node api error, but after i killed the process and reopened the app, the cpu inference worked fine. Thats a small oversight i didnt take into account where llamacpp is still doing stuff while im trying to kill the node process.

Well, thats it! See you next time in “trying to make lobotomy ai useful so people stop using giant datacenters to ask about how long to boil and egg for”