Tokenization

A model does not see your prompt as words or characters. It sees token ids produced by a tokenizer. That is why a short-looking string can be expensive, a long …