Simple invisible-character stripping can break emoji while letting hidden LLM prompts through
A technical analysis warns that naively removing invisible Unicode characters from text can destroy legitimate emoji — such as family sequences built with Zero Width Joiner characters — while failing to strip hidden instructions embedded via Unicode tag characters. The article identifies four distinct families of invisible code points, each with different legitimate uses and different security risks, arguing that treating them as a single category leads to both over-removal and under-removal. Unicode defines a formal set called Default_Ignorable_Code_Points, covering 4,174 characters, which the author uses as the correct detection baseline rather than short hand-written regex patterns that typically cover only a handful of these. The recommended approach assigns a separate policy to each family — keeping joiners and bidi controls in prose, but removing tag characters entirely, as they have no legitimate rendering use and can carry hidden text instructions to LLM applications. The piece also notes that visually blank but width-bearing spaces like U+202F, reportedly appearing in OpenAI model output, fall outside this set and require a separate whitespace normalisation policy.
