Tokenizer done now for vector context calculator . The tokenizer was somwhat refusing to work but one rewrite later it subdued to my power it is split into teo parts the text decoder and encoder decoder formats the words for encoder and encoder finds tokens for the words