groq rate limit almost killed my whole app

so i built randomgenerals.com. its an ai assistant, chat + code + image
gen + diagrams. runs on a free oracle vm, 2 arm cores no gpu. flask app
that does the website, the pwa and the backend for the desktop version.
accounts, credits, emails, dashboard, all of it.

ok so the bug.

groq free tier is 8000 tokens per minute. not per request. per minute.
for the entire key. every single user on my site sharing that. and it
counts your prompt AND the reply you asked for as one number together.

i wasnt checking that anywhere lol

so if a chat got long enough (like 5400 tokens) it would just stop
working. forever. you hit retry and it builds the same too big request
and gets refused again. same error every time. only way out was starting
a new chat and nobody knows to do that.

my first idea was just make the replies shorter. bad idea. thats the
thing people actually came for. so now it drops old messages first and
only shortens the reply if that wasnt enough. forgetting what you said 6
messages ago is annoying. not getting an answer is worse.

then i found another one. i was never checking finish_reason so when a
reply hit the token limit it just stopped mid sentence and looked
completely normal. and the api returned 200 every time so my logs were
clean. no clue how long that had been happening ngl

126 commits so far. randomgenerals.com