A token is the unit of text an AI language model actually works with. Models do not read letter by letter, and they do not read whole sentences. They break text into tokens, which are words or fragments of words, and everything they do is expressed in those units.
As a rough rule for English, one token is about four characters, and 100 tokens is roughly 75 words. A common word like "bank" is a single token. A longer or unusual word like "diversification" splits into several.
Why models tokenise at all
A model needs a fixed vocabulary of symbols it can predict from. Using whole words would require a vocabulary of millions and would break on any word it had never seen. Using individual letters would work but would make sequences impossibly long and strip away meaning.
Sub-word tokens are the compromise. Common words get their own token, rare words are assembled from pieces, and the model can therefore handle a word it has never encountered by building it from fragments.
Rough conversions worth remembering
| Text | Approximate tokens |
|---|---|
| One common English word | 1 |
| The phrase "AI is useful" | About 4 |
| A short paragraph, 100 words | About 130 |
| A 500 word document | About 650 to 700 |
| A 10 page PDF | Roughly 5,000 to 7,000 |
| A 300 page book | Roughly 100,000 or more |
Why tokens cost more in Indian languages
This is a real and underappreciated issue.
Tokenisers are trained mostly on English text, so English is encoded efficiently. Hindi, Tamil, Bengali, Telugu, Marathi and other Indian languages written in their own scripts are frequently split into far more tokens for the same meaning, sometimes several times as many.
Two practical consequences follow. On a paid API, the same message in Hindi can cost noticeably more than in English. And because the context window is measured in tokens, a Tamil document fills the available space faster than an English one of the same length. Indian language models being built domestically are addressing exactly this by training tokenisers on Indian scripts.
The two things tokens actually govern
The context window. A model can only hold a limited number of tokens in view at once, and this budget covers your prompt, any documents you paste, the conversation so far, and the reply it is generating. Modern models offer large windows, but in a long conversation or with a big document, earlier content can drop out of view. If a chatbot seems to have forgotten something you said an hour ago, this is usually why.
The price. Paid APIs charge per token, separately for input and output, with output usually costing more. If you are building anything on top of an AI API, token count is your cost model. A chatbot that resends the entire conversation history with every message is paying for that history repeatedly.
For ordinary users on a monthly subscription, tokens are invisible. For anyone building a product or automating a workflow, they are the main line item.
Practical implications
- Long documents may need splitting. If a PDF exceeds the context window, break it into sections and summarise each, then combine the summaries.
- Trim what you paste. Sending only the relevant pages rather than an entire annual report reduces cost and often improves the answer, since the model has less irrelevant material to weigh.
- Long conversations degrade. Starting a fresh chat for a new task usually produces better results than continuing a thread that has drifted across five topics.
- Output limits are separate. A model may accept a very large input but cap how much it will generate in one response, which is why asking for a 10,000 word document in one go often produces something shorter.
Tokens are also the mechanism behind how a large language model generates text: it predicts one token at a time, appends it, and repeats. That is why responses stream in gradually rather than appearing all at once.