The soft hyphen (U+00AD): the invisible character that breaks PDF text extraction

The character that hides in plain sight

Both independent AI reviews of the cover PDF surfaced the same critical finding. The extracted text contained anomalous characters between syllables: pre￾mier, héri￾tier, trans￾mission. The character appearing between those syllables is not a hyphen, a dash, or a visible separator of any kind. It’s U+00AD - the soft hyphen.

On screen: invisible. In print: invisible. When text is extracted: ￾.

The soft hyphen — the PDF’s wrench in the works

What a soft hyphen is

Unicode U+00AD is a conditional line-break hint. It tells a rendering system: “if you need to break this word at a line end, break it here and render a hyphen.” If no break is needed, the character is invisible - it takes no space, it renders nothing, it’s simply not there.

In running body text, this works correctly. The word premier sits on one line, the soft hyphen is ignored, and nobody knows it’s there. If the word is at a line end and a break is needed, the renderer breaks the line, inserts a hyphen, and moves mier to the next line. This is what typesetting engines do with soft hyphens.

The problem arises at text extraction. PDF text extraction doesn’t render lines - it reads the character stream from the PDF’s internal content encoding. A soft hyphen in the character stream is a character. It has a Unicode code point. Text extraction faithfully reports it: U+00AD. Depending on the character set and rendering context, this appears as a visible glyph - typically a broken rectangle ￾ - between syllables.

Where it comes from

Soft hyphens in a PDF source can arrive from several places:

Word processor hyphenation. Microsoft Word sometimes inserts soft hyphens when automatic hyphenation is enabled and a word is manually hyphenated by the user with Ctrl+- (Windows) or Option+- (Mac). When the Word document is exported to another format, the soft hyphen may survive.

OCR and transcription pipelines. Optical character recognition and some transcription tools insert soft hyphens as part of line-break reconstruction. If a transcribed document was formatted for a narrow column, hyphenated words from the original layout may have been transcribed with embedded soft hyphens.

Direct Unicode insertion. Someone typed U+00AD directly into a source file, possibly intending it as a regular hyphen or as a typographic hint that was appropriate in a different rendering context.

For a pipeline that converts Word transcriptions to LaTeX, the first and second sources are both plausible. The transcription pipeline needs to strip soft hyphens from the Word source before they propagate to LaTeX and from LaTeX into the PDF.

The character is invisible everywhere a human looks and only appears when the PDF is processed - search, copy-paste, accessibility checks. The fix is one normalization step, applied before the character can propagate into LaTeX.

How to find it

In a text file or source document:

grep -rn $'\xc2\xad' src/

U+00AD encodes as 0xC2 0xAD in UTF-8. The grep above finds every file in the source tree that contains the character and reports the line number.

In a PDF:

Most PDF viewers won’t show the soft hyphen directly. Copy a block of text from the suspect PDF and paste it into a plain-text editor. If a stray U+00AD turns up between syllables - rendered by the editor as a broken box �, a thin gap, or nothing at all - the soft hyphen is present. A hex view (xxd) is the unambiguous check: look for the bytes c2 ad.

An accessibility checker - PDF/UA validators, Adobe Acrobat’s accessibility tools - will also flag embedded soft hyphens as anomalous characters in the character stream.

In a LaTeX source:

If the soft hyphen entered via the Word-to-LaTeX conversion step, it may appear as \- (LaTeX’s soft hyphen command) or as a literal U+00AD in the .tex file. Both grep commands are useful: grep for the Unicode code point and for \-.

How to fix it

In the Python preprocessing pipeline, add a normalization step before LaTeX conversion:

import re

def strip_soft_hyphens(text: str) -> str:
    return text.replace('\u00ad', '')   # explicit escape, not an invisible literal

Apply this to all text extracted from Word documents before it enters the LaTeX pipeline. The soft hyphen carries no semantic content in typeset text - it’s a rendering hint, and LaTeX’s own hyphenation engine will handle line breaks without it. Write the character as '\u00ad', never as a pasted literal: an invisible character sitting in source is the same hazard you’re trying to remove, and it won’t survive a copy-paste or a careless editor intact.

If soft hyphens appear in the LaTeX source directly (as \-), they’re more ambiguous - \- is sometimes used for legitimate manual hyphenation control in LaTeX. Review each instance: if it was carried over from the transcription pipeline, remove it; if it was added deliberately for a specific word’s line-breaking behavior, leave it.

The broader lesson

The soft hyphen is an example of a class of problem that’s invisible at production time and surfaces in post-production or deployment. The PDF looks correct. It prints correctly. The error only appears when the artifact is processed in an unexpected way - text extraction, accessibility scanning, search indexing.

For a print-primary project, “it looks right and prints right” is usually sufficient. For a project where the PDF will also be distributed digitally, searched, or submitted to accessibility review, the invisible character matters. The fix is cheap - one normalization step in the pipeline - but only if you know to look for it.

This is also the smallest possible preview of a larger obligation. The same character stream that a search index reads is the one a screen reader reads, and “it looks right” says nothing about whether the document is accessible - whether it carries a tag tree, a reading order, a declared language, and alternative text. Accessibility is no longer a nicety a print shop can ignore; regulation and procurement increasingly require it, and the field treats it as a core responsibility rather than a late add-on. The soft hyphen is the canary: if a single invisible code point can corrupt extracted text, the rest of the accessibility layer deserves the same build-from-the-source discipline, which is the subject of tagged from the source.

AI-assisted review found it. The human reviewer, having stared at the same PDF for a week, did not.

Summing up

1. The soft hyphen hides in plain sight. U+00AD is invisible on screen and in print but surfaces as a broken box ￾ between syllables when text is extracted.
2. It is a conditional break hint, not a character with content. A renderer shows a hyphen only if it needs to break the word; otherwise the code point sits silently in the stream.
3. The damage shows at extraction. Accessibility checkers, search indexing, and copy-paste read the character stream, not the rendered line - and report the soft hyphen faithfully.
4. It arrives from a few sources. Word manual hyphenation, OCR/transcription line-break reconstruction, or direct Unicode insertion - all plausible for a Word-to-LaTeX pipeline.
5. Find it by its bytes. U+00AD is 0xC2 0xAD in UTF-8: grep -rn $'\xc2\xad' in source, an xxd check for c2 ad in a PDF.
6. Fix it with one normalization step. Strip '­' (written as an escape, never a pasted literal) before LaTeX conversion - LaTeX’s own engine handles line breaks.
7. It is a class of bug, not a one-off. Invisible at production, surfacing only when the artifact is processed unexpectedly - cheap to fix, but only if you know to look.

External sources

• What U+00AD is and how it behaves
• Unicode code-point reference for U+00AD
• Distinguishing soft hyphen from real hyphens

Hero image: “Cracked Plaster Wall” by J P Davidson, licensed under CC BY 2.0.