Text guides

Why two strings that look identical do not match

A lookup fails, a duplicate slips through, a join returns nothing. Four invisible differences cause almost all of it, and they are worth checking in a fixed order.

Reviewed and updated

A lookup returns nothing. A deduplication pass leaves two rows that read the same. A join that should match four hundred records matches three hundred and ninety one. Every one of these has the same shape: two pieces of text that a person reads as identical, and a computer reads as different.

There are four common causes. They are all invisible on screen, they have different fixes, and checking them in the wrong order wastes more time than any of them individually.

First: whitespace at the ends

This is the most common cause by a wide margin, and the easiest to rule out. A trailing space survives a copy and paste, a CSV export, a form submission and a database import without ever being visible. Leading spaces arrive from indentation in the source, and from spreadsheet cells where somebody aligned a column by typing.

The reason to check it first is that it costs nothing to check. Compare the character counts of the two values. If they differ by one or two and the text reads the same, you have found it.

Trimming both ends of every line fixes it. The related decision, once you are trimming, is what to do with lines that hold nothing but whitespace: those are blank lines to a reader, and a tool that tests for emptiness before trimming leaves every one of them behind as an invisible row.

Second: the same letter stored two ways

This one surprises people who have never had to think about Unicode, and it is the reason a name can fail to match itself.

An accented letter can be stored in two different ways. The letter e with an acute accent can be a single code point, or it can be a plain e followed by a separate combining accent character. Both render identically. Neither is wrong. macOS has historically favoured the decomposed form and Windows the composed one, so a file that has passed through both can contain both spellings of what a reader sees as one name.

No comparison treats them as equal, because the byte sequences genuinely differ. The fix is to normalise both sides to the same form before comparing, or, when the destination only accepts ASCII, to remove the accents entirely.

Removing accents has its own trap. Normalisation splits a letter into a base plus its marks, so dropping the marks handles most of Latin script. It does nothing at all for letters that are not a base plus a mark: the German eszett, the Scandinavian o-slash and ash, the Polish l-stroke, the Icelandic thorn. A tool built on normalisation alone reports success and hands those back unchanged, which means the one record that still fails to match is the one belonging to somebody whose name contains them.

Third: tabs against spaces

In anything structured by indentation, a tab and a run of spaces look the same and are not. This breaks diffs, which show a whole block as changed when only the indentation character differs, and it breaks any comparison of the lines themselves.

Converting between them is worth understanding rather than doing blindly. A tab does not mean a fixed number of spaces. It means advance to the next tab stop, so a tab at the start of a line is worth the full width and a tab after two characters is worth only what remains. Replacing every tab with four spaces produces a file that disagrees with the original about where the columns sit, which is usually the exact problem the conversion was meant to solve.

Fourth: characters carried in from a rendered page

Text copied out of a web page or an email client brings things with it. Non-breaking spaces, used to stop a line breaking between a number and its unit, are the most common. Soft hyphens, zero-width joiners and directional marks all appear. Each is a real character that a comparison counts.

If the text came from markup rather than from a rendered page, the same problem arrives in a larger form: tags, entities, and the contents of script and style elements that a naive strip leaves behind as words.

These are harder to find than the first three, because they can sit anywhere in the string rather than only at the ends, and because most of them have no visible width at all. The practical test is a length comparison against a version of the same text you know to be clean. If the count is higher than the number of characters you can see, something in there is invisible, and the position of the difference tells you where to look.

A related case is worth naming because it looks like a different problem entirely. Text pasted out of a spreadsheet arrives tab separated, and text pasted out of a rendered table arrives space separated, so the same table copied from two places produces two files that no comparison will reconcile until the separators are made the same.

The order is the point

Check whitespace, then normalisation, then tabs, then everything else. That order runs cheapest first, and each step changes what the next one can see: normalising a string that still has a trailing tab tells you very little, and comparing lengths after the invisible characters are gone is a different measurement from comparing them before.

The FileGizmo way

Free tools. Never uploaded.

Good to know

Frequently asked questions

How can I see the difference if it is invisible?

Compare the lengths first. Two strings that display identically but report different character counts differ in something you cannot see, and the size of the difference usually names the cause. One character is often a trailing space, while a difference matching the number of accented letters points at normalisation.

Why does the same name sort differently in two systems?

Usually normalisation. A letter stored as a base plus a combining mark sorts differently from the same letter stored as a single code point, because the byte sequences differ even though the rendering does not.

Is a non-breaking space really different from a space?

Yes. It is a different character with a different code point, and every comparison treats it as such. It arrives constantly in text copied out of a rendered web page, where it was used to stop a line breaking in an awkward place.

Should I normalise before or after trimming?

Trim first. Normalisation can change the length of a string, and a trailing non-breaking space is easier to reason about once the ordinary whitespace has already gone.