Chatbot RAG
By
@Hamna
- 10 Devlogs
- 72 Total hours
It's a AI RAG chatbot which mimics my friend's personality using Llama fine tuned with LoRA
Ship #2
Made a RAG chatbot that mimics my friends personality using Llama and Hugging Face.
NOTE: I’m using zero GPU on HF so after 100 msgs it goes to sleeping mood , in that case the chatbot would reply a random thing like “system off” , “internet is down” or something like that , so don’t worry about that, come after 24Hr and she’ll talk normally
FACT: Senpai used to say a lot “Haii haii” which is a japanese phrase for “yes” so don’t worry about that either, she’s just being friendly with ya
Hope ya’ll love this version of chatbot
32m 47s logged
Guyss!! This is gonna be the last change I’ve made today, Senpai chatbot is live now and she’s talking like my friend (really couldn’t be more happy to see her talk to me again 🥹 )
25m 47s logged
Made some Minor UI changes on frontend nextjs
9h 36m 12s logged
After testing the chatbot with around 50 beta users, I found several issues that weren’t obvious during development. Responses were too long, the personality became too generic when using RAG, Urdu and English were sometimes mixed incorrectly, and the mobile layout needed work. I shortened the responses, restricted web search to things like news and sports, improved the system prompt and input handling, redesigned the mobile UI, and added a message explaining the cold start delay
8h 28m 36s logged
I added logging, I started finding problems that weren’t obvious from the frontend. Web searches were sometimes failing silently, too many messages were causing token overflows, and using both max_length and max_new_tokens was causing unexpected behavior. I fixed these issues by adding proper error handling, limiting the conversation context, and simplifying the token settings. I also built a small debug dashboard to track searches, response times, token usage, and errors (Ughh I’m really exhausted now)
8h 47m 50s logged
I initially moved the deployment to Replicate to improve response speed. It worked and responses dropped from around 15–30 seconds to about 5–10 seconds but then it also introduced extra cost and complexity. In the end, I decided to stay with Hugging Face Spaces because the free setup is more suitable for this side project. Also improved the HF app.py file
8h 34m 24s logged
I noticed that the model was giving very generic and robotic responses even though it worked much better during training. After debugging, I realized I was using a different system prompt during inference than the one I used while fine tuning. Once I switched back to the exact same prompt, the responses became much more consistent. Also fixed Hackclub key errors , now it’s working fine
Ship #1
I built a AI RAG chatbot that mimics my friends personality , trained on whatsapp chats so it actually talks like my friend instead of generic robot ai. fine tuned llama 3b with LoRA on kaggle’s free gpu, deployed it on hf spaces zeroGPU gradio for backend and next.js on vercel for frontend, plus added RAG so it can clean up messy texts and pull live info from the web when needed. it genuinely feels like talking to my friend, all built for free. to test it. just start chatting, first msg might take a sec to load (model’s waking up)
10h 9m 17s logged
wanted senpai to be more. Previous It was only talking with using
fine tuned Llama model but if i’ll ask it something like what’s
happening in the world, it won’t talk back like my friend cause it
didn’t have that latest information of the world so i added RAG. used
the hackclub api as a middleman llm that cleans up messy input and
decides if a web search is needed ( like “what’s the fifa score rn” this
will be flagged as a web search needed query)
two bugs along the way:
responses were so slow at first , turns out just zeroGPU cold starts, chills out after msg 1
model started sounding fake n robotic bc i overwrote it with a huge system prompt. fixed this by making the heavy prompt short
so now gng it’s live, plz talk to her
7h 47m 18s logged
I uploaded my model on Hugging Face with zero GPU
(made a new hf space called amadeus, picked gradio as the sdk, zeroGPU as hardware)
If someone else is thinking about starting in HF, just wanna tell the annoying part: cold starts. first message takes 20-40s while the model loads into memory. after that it’s quick like 3-8s
then built an actual next.js frontend since raw gradio ui felt kinda mid, hooked it up through an api route, deployed on vercel with my env keys
And it’s live
11h 17m 11s logged
Trained the data ( This took soo much time fr)
I used kaggle’s free gpu (t4, sometimes p100 if i got lucky) to fine tune llama 3b with LoRA through PEFT.
quick explainer since people asked LoRA freezes the whole base model (32m params untouched) and only trains a tiny set of adapter params (420k). instead of retraining the entire model I put a extra layer of changed weights on top of the model which is way cheaper and faster than the traditional training
tested it after training and it WORKED. The model is giving the same energy, same little phrases, actual personality of my friend which is kinda scary ngl
notebook’s public if you wanna nerd out: https://www.kaggle.com/code/hamnamubarak/amadeus/edit
6h 23m 10s logged
so i’m building senpai, an ai that actually talks like my friend, not some normal chatbot. so first thing first, I need data to learn from so here’s wht I did:
exported a whole year of whatsapp chats with my friend (yes i read through all of it, and got his consent). wrote a python script on kaggle to clean it up and formatting it into proper training pairs
not glamorous but this is like 30% of the actual work tbh, garbage data can cause strange and irrelevant replies
Then I trained this on Kaggle with Llama model with LoRA adaptation
Right now the replies aren’t so great, I’ll do some fine tuning later
