Hidden characters: what the terminal sees that you cannot

Short answer: paste your text into the box below and it tells you whether there are any, how many, and exactly which code point each one is. A hidden character is an ordinary Unicode code point that paints no ink — U+200B, U+00A0, U+FEFF, U+200D and a dozen more. Your editor draws nothing for it, your editor’s search will not match it, and diff will happily print two lines that look byte-for-byte identical on screen and tell you they differ. Nothing here is uploaded; the scan runs in your browser.

The box answers one question — is anything there. When the text lives in a file on disk rather than in your clipboard, the rest of this page is what to run instead, and what each command actually prints. Every number below was measured on this machine against files built from a single known code point, so the outputs are reproducible rather than remembered.

Nine invisible characters, nine different strings

The most useful command is also the oldest one. cat -A does not interpret the bytes as text at all: it walks them and renders anything outside printable ASCII in a notation you can read. That is what makes it better than an editor with a “show invisible characters” setting, because the editor decides what to draw and the notation does not.

Measured, one file per character, each holding Hello + the character + World:

Two rows are worth staring at. The no-break space comes out as M-BM- followed by an actual space, and the soft hyphen comes out as M-BM-- — a hyphen. Both look like content you typed. If you are skimming cat -A output for something unusual, those two are the ones that slide past, even though the notation is technically telling you the truth.

The other thing the table shows is that these characters are not interchangeable. The Hangul filler is the only one that adds two columns of display width, the no-break space and the line separator add one, and the remaining six add nothing at all. Width is a fingerprint: it narrows down which one you are holding before you have identified it.

The cleanup pattern everybody recommends misses three of them

The standard advice for removing invisible characters is a character class built on the Unicode “other” category, \p{C}, or its format subcategory, \p{Cf}. It is in the accepted answers, it is in the sanitising snippets, and it is wrong at the edges.

Measured with grep -P against all nine files above:

Six of nine, twice. The three that survive are the no-break space and the line separator, which are separators rather than format characters, and the Hangul filler, which is a letter with a blank glyph and matches no “other” class at all. Those are not obscure choices: the no-break space is the one people paste by accident most often, and the Hangul filler is what most blank-name generators hand out.

The third row is the boring one and the one that works. Sweeping for anything outside printable ASCII and tab matched nine of nine and returned nothing on a clean control file. That is the minority position — the advice usually given is to declare what you will accept rather than enumerate what you will reject — and on this evidence the minority is right, because an allow-list cannot be surprised by a character nobody thought to put on the deny-list.

It is also worth saying what that sweep costs you: it will flag every accented letter and every CJK character in the file. It is the right pattern for identifiers, config keys and filenames. It is the wrong one for prose.

Two counters, one line, different answers

People reach for wc -m to find out how many characters a file holds, and it is usually right. It was wrong here.

A line of Hello + U+E0020 + World is twelve code points. wc -m reported thirteen. awk '{print length($0)}' reported eleven, which is correct once you take the newline off. The same thing happened with an emoji: a four-code-point line was reported as five. A line of plain ASCII was reported correctly at four.

The pattern is the cause. On a shell where the wide character type is sixteen bits, a code point outside the basic multilingual plane needs two of them, and wc -m counts the storage units rather than the characters. So a single tag character inflates the count by one, and so does every emoji, and nothing in the basic plane does. If you are using wc -m to sanity-check a length, it agrees with reality only until somebody pastes something colourful into the file.

The same command disagrees with itself in a different way. wc -w counted two words for the no-break space and the line separator, and one for the other seven — the same one word as a control file with no character in it at all. So wc considers a no-break space a word boundary and a zero-width space not a word boundary, which is defensible, and it also considers a line separator a word boundary while wc -l refuses to count it as a line break, which is harder to defend.

Finding the position instead of counting

Counting tells you something is there. To find out where, stop treating the file as text.

cmp compares bytes and gives you the offset directly. Against a file holding Hello + U+200B + World and a clean HelloWorld, it reported differ: byte 6, line 1. Six is the first byte of the three-byte character, which is the answer you wanted and the answer no text-based tool will give you.

diff on the same pair prints this:

1c1
< HelloWorld
---
> HelloWorld

Two lines that render identically, separated by a marker saying they are not equal. This is the moment most people conclude the tool is broken. It is not; the bytes differ and the terminal is drawing the zero-width space as nothing, because that is what it is.

De-duplication has the same problem from the other side. A file with Hello + U+200B + World on one line and a plain HelloWorld on the next stays at two lines under uniq and under sort -u. Two copies of the invisible version collapse to one, exactly as they should. So the tools are consistent, and both results look wrong on screen.

Searching: why your grep comes back empty

The failure everyone hits first. A file holds Hello, a zero-width space, and World. Searching for HelloWorld returns zero matches. Searching for Hello returns one. The string you can see is not the string in the file, and the pattern is matching the file.

To search for the character rather than around it, give grep the code point:

grep -P '\x{200B}' file.txt

Measured, that matches the file with the character and returns nothing on the clean one. The same form works for the no-break space with \x{00A0} and for the tag characters with \x{E0020}, so non-BMP code points are not a problem. What you cannot do is type the character into the pattern, because your shell and your editor will both happily store something you cannot check.

Note the -P. Without it, the escape is not interpreted and the pattern matches nothing — a second way to get an empty result that looks identical to the first.

The conversion that eats the rest of your line

This one is worth knowing before you run it on anything you care about. Transliterating a file down to ASCII sounds like a safe way to drop invisible characters, and GNU iconv does not skip what it cannot convert — it stops.

Measured on the same Hello + U+200B + World file, iconv -f UTF-8 -t ASCII//TRANSLIT printed Hello and nothing else, then exited with an error at position 5. World was gone. The command did not remove the invisible character from the middle of the line; it emitted everything up to the character and abandoned the rest.

Adding -c, which omits unconvertible characters instead of failing, produced the full HelloWorld and exited cleanly. If an ASCII conversion of a file comes out short, this is usually why, and the exit status is the only warning you get.

The older approach still works and is safer for a one-off: hand tr -d the three octal bytes of the character rather than the character itself.

tr -d '\342\200\213' < file.txt

That removes U+200B and leaves everything else alone. The bytes are what U+200B looks like in UTF-8, which is also what od -c will show you if you do not know them: 342 200 213.

Where they come from, and why some of them are deliberate

Three sources account for nearly all of it. Editors and word processors insert no-break spaces and zero-width spaces as you type, and they travel with the text when you copy it. Something saved a file with a signature and left a byte-order mark at the front — the three bytes EF BB BF, which is why it turns up as %EF%BB%BF when a URL gets copied out of a browser. And text that came out of a chat window, a document or a web page carries whatever that source was using.

Not all of it is accidental, and this is the part that makes a blanket strip dangerous. A zero-width space in the middle of a domain name is a well-known way to stop it being turned into a shortened link, and it is also a documented way to get a URL past a link scanner while every major browser still renders it clickable. People put these characters in on purpose. If you are looking at a file and find one, the honest answer is that you cannot tell from the character alone whether it was meant to be there.

The byte-order mark is the one case where position tells you something: it is at the very front, it is three bytes, and cat -A renders it as M-oM-;M-?. The rest can be anywhere — in the middle of a word, between two words, or at the end of a line — which is what makes the “it is always the first character in the file” rule that gets repeated for this problem so misleading. That rule is true of the byte-order mark and of nothing else.

Taking them out

For a pasted string, the zero-width space remover marks each one in place before anything is deleted, and the invisible Unicode list has a scan that reports counts by code point. For a file, tr -d with the octal bytes is the blunt instrument and grep -P with the code point is the way to confirm you got them all. Run the scan again after — that is the only check that means anything.

Leave the joiner alone unless you have looked at what surrounds it. U+200D is load-bearing in an emoji sequence, and deleting it does not clean text up, it breaks a picture somebody chose. See what a zero-width joiner does before you strip one.

What this page does not tell you

Whether the character you found was put there deliberately. Some are, and nothing in the bytes distinguishes an accident from an intention.

It also does not tell you which of these your own terminal will draw. The cat -A strings above are GNU coreutils output; a different implementation may render the same bytes in a different notation, and the wc -L widths come from that implementation’s idea of display width. The code points and byte counts are stable. The notation is not.

Measured 2026-09-27 with GNU coreutils and GNU grep 3.0 on Windows via Git Bash, cross-checked with Node 22.22.2 (ICU 78.2, Unicode 17.0) and Python 3.13.14. Related: the full list of invisible characters, what one does to a form field, tag characters that hide a whole ASCII message, U+2028 and U+2029 in JSON, why there is no empty character, and the JSON cleaner if it is a payload that will not parse.