In French, the same prompt costs more tokens
· updated on
What a token is, why the tokenizer splits French into more pieces, and what it costs: on a long text and its official translation, French needs 1.14 to 1.50 times more tokens depending on the tokenizer. Accents barely matter, and a customer service chatbot pays 17% more.