Tokens and Context Windows

What is a Token?

LLMs don't read text character by character or word by word — they read tokens. A token is a chunk of text, typically 3-4 characters or about ¾ of a word.

"Hello, world!"  →  ["Hello", ",", " world", "!"]       = 4 tokens
"unbelievable"   →  ["un", "believ", "able"]             = 3 tokens
"ChatGPT"        →  ["Chat", "G", "PT"]                  = 3 tokens
"a"              →  ["a"]                                 = 1 token

Rule of thumb: 1 token ≈ 4 characters ≈ ¾ of a word. 100 tokens ≈ 75 words.

Why Tokenization Matters

Everything in an LLM is measured in tokens:

• Cost — you pay per token (input + output)
• Speed — latency scales with token count
• Limits — context window is measured in tokens

Token Costs

Different models charge different rates per token:

Output tokens cost more — generating text is more expensive than reading it.

What is a Context Window?

The context window is the maximum number of tokens an LLM can process at once — both input and output combined.

Everything — system prompt, conversation history, retrieved documents, user message, and the model's response — must fit within the context window.

Context Window Sizes

What Happens When You Exceed the Context Window?

Most production systems use a combination — summarize old conversation history and use RAG to retrieve only relevant documents rather than stuffing everything in.

Counting Tokens Before Sending

Always count tokens before making an API call to avoid errors and control costs:

import tiktoken  # OpenAI's tokenizer
 
def count_tokens(text: str, model: str = "gpt-4o") -> int:
    enc = tiktoken.encoding_for_model(model)
    return len(enc.encode(text))
 
# Check before sending
prompt = "Explain quantum computing in simple terms"
tokens = count_tokens(prompt)
print(f"Prompt uses {tokens} tokens")  # Prompt uses 8 tokens
 
# Estimate cost
cost = (tokens / 1_000_000) * 2.50  # GPT-4o input rate
print(f"Estimated cost: ${cost:.6f}")

Key Takeaway

• 1 token ≈ 4 characters ≈ ¾ of a word
• You pay per token — both input and output
• Context window = maximum tokens the model can see at once
• Output tokens cost more than input tokens
• Always count tokens before sending to avoid surprises
• When context is too large: truncate, summarize, or use RAG