With chatgpt help i optimized my tokenizer. 100MiB needs only 33.72s of training time. Batch encode throughput is around ~400MiB/s which is around 11.5x faster than OpenAI tiktoken(look at images). I have CPU AMD 5 5600H inside PC. I think that on AI data center my tokenizer could reach even about 4GiB/s ! My tokenizer is fast but it require relative big amount of ram. Nedded RAM is about 20x of train file size. When I finish my project I will train my model and benchmark my tokenizer on god tier machine. Wish me luck!