Worked over a LOT of stuff, like doing apples to apples comparisions of models such as LLama 11b vision instruct, So over time I fixed a lot of stuff

added a new profile, that quants and quants and quants all layers till the model fits your VRAM, it doesnt care if the model is 20gb in weigths and you have a 8gb vram, it will quant and make it fit

So testing this autofit profile on LLama 11b vision instruct, heres what i found

it hit a cosine of 0.939758 and around 24tk/s on a 200 token test

Similarly i tested ollama Llama 11b vision instruct with q4_k_m quant, it got a cosine close to 0.720495 i would say it would be more closer to 0.8 because this cosine was for only top 20 logprobes that ollama exposes

This was done on
8gb vram rtx 5050 laptop
32gb ddr 5
ryzen 7 250
arch linux

and the comparision is apples to apples because the thinruntime also quanted the model aggressively