What invisible characters cost you in LLM tokens

Short answer: about one token each. A 1,000-word text with a single zero-width space after every word goes from 1,103 tokens to 2,103 — a 90.7% increase — before you have added a single word of meaning. The characters are invisible on screen, but the tokenizer sees every one of them and you pay for every one of them.

Measured, not estimated

These are the numbers from a 1,000-word English sample, tokenized twice — once with o200k_base and once with cl100k_base, the encodings behind current OpenAI models. Only difference between rows: invisible characters inserted at a fixed interval.

The pattern is close to linear: one invisible character in, roughly one token out. Work out the cost of any piece of text by counting the hidden characters — the token count rises by about that much.

Why a character you cannot see still costs a token

Tokenizers are built from byte-pair merges learned on ordinary text. Common English words collapse into single tokens. Zero-width and invisible characters are rare in training data, so they never earned a merge of their own — the tokenizer falls back to splitting them into their raw UTF-8 bytes, and each of those bytes becomes a token. U+200B is three bytes in UTF-8 (E2 80 8B), which is why one invisible space lands you roughly one extra token rather than zero.

That is the whole mechanism. There is nothing exotic about it, which is exactly why it gets missed: everything about these characters says they are not there.

Where this actually bites

• Pasting model output back into a model. The most common way to carry a watermark forward is to feed yesterday's output into today's prompt. You pay for the hidden characters on every round trip.
• RAG pipelines. Chunks scraped from pages or documents that carry invisible characters inflate every retrieval, and the inflation is invisible in your usage dashboard because the text looks short.
• Batch jobs. A 90% token increase on a nightly job over a large corpus is not a rounding error, and it does not show up as an error at all.
• Context limits. Tokens spent on characters that carry no meaning are tokens unavailable for the content you actually wanted in the window.

Check your own text

Count the invisible characters first, then multiply. The zero-width space remover shows you where they sit and how many there are; the invisible characters list has the code points if you want to look them up by hand. If you are cleaning a payload rather than prose, the JSON cleaner and Excel cell cleaner do the same job on structured input.

Reproduce it

The measurement above is three lines of Python. Run it on your own text rather than trusting a table on someone else's site:

import tiktoken
enc = tiktoken.get_encoding("o200k_base")   # or "cl100k_base"
dirty = "Your text here"                    # paste as-is, do not clean it
clean  = dirty.replace("\u200b", "").replace("\u200c", "") \
                .replace("\u200d", "").replace("\ufeff", "")
print(len(enc.encode(dirty)), len(enc.encode(clean)),
      len(enc.encode(dirty)) - len(enc.encode(clean)))

The third number is what the hidden characters are costing you per call. Multiply it by your request volume and your provider's rate.

What this does not tell you

This page measures cost, not authorship. Extra tokens tell you characters are present; they do not tell you who put them there or why, and they are not evidence that a text was generated by a model. Token counts also vary by encoding, so the percentages above apply to the two encodings tested — run the snippet against your own model's tokenizer if you need a number you can bill against.

Measured 2026-09-21 with tiktoken on a 1,000-word English sample. Tools: Claude watermark remover, zero-width space remover, JSON cleaner, Excel cell cleaner, invisible characters list.