Non-Printable Characters

Most applications and programming languages do not support any special syntax in the replacement text to make it easier to enter non-printable characters. If you are the end user of an application, that means you’ll have to use an application such as the Windows Character Map to help you enter characters that you cannot type on your keyboard. If you are programming, you can specify the replacement text as a string constant in your source code. Then you can use the syntax for string constants in your programming language to specify non-printable characters.

The Just Great Software applications are an exception. They allow you to use special escape sequences to enter a few common control characters. Use \t to replace with a tab character (ASCII 0x09), \r for carriage return (0x0D), and \n for line feed (0x0A). Remember that Windows text files use \r\n to terminate lines, while UNIX text files use \n.

Python and Boost support the above escape sequences in replacement text too. They also support these more exotic non-printables in replacement text: \a (bell, 0x07), \f (form feed, 0x0C) and \v (vertical tab, 0x0B).

ICU supports hexadecimal escapes to insert Unicode code points. For BMP characters you can use \uFFFF with 4 hexadecimal digits. For astral characters you can use \U0010FFFF with 8 hexadecimal digits representing a value up to 10FFFF. \u20AC inserts the euro currency symbol, for example.

Boost supports hexadecimal escapes. PCRE2 supports them if you select its “extended” replacement string flavor. If you use Boost or PCRE2 with 8-bit characters then you can use \x00 through \xFF and \x{0} through \x{FF} to insert any character from the 8-bit code page. If you are using Boost with wide characters or PCRE2 with UTF-8, UTF-16, or UTF-32 then these insert a code point between U+0000 and U+00FF. Boost with wide characters supports \x{FFFF} to insert any code point up to U+FFFF. PCRE2 using UTF supports \x{10FFFF} to insert any Unicode code point.

The Just Great Software applications support \uFFFF with 4 hex digits to insert Unicode code points up to U+FFFF. JGsoft V2 adds \x{FFFF} with up to 4 hex digits as an alternative syntax. These escapes always represent Unicode code points, regardless of the file’s encoding. The JGsoft apps also support \x00 through \xFF to insert a byte. When EditPad or PowerGREP process a file using an 8-bit code page then this byte selects a character from that code page. When they process a Unicode file, the byte is interpreted as a Unicode code point between U+0000 and U+00FF.

Python does not support hexadecimal escapes in the replacement text syntax, even though it supports \xFF and \uFFFF in string constants.

Regex Syntax versus String Syntax

Many programming languages support escapes for non-printable characters in their syntax for literal strings in source code. Then such escapes are translated by the compiler into their actual characters before the string is passed to the search-and-replace function. If the search-and-replace function does not support the same escapes, this can cause an apparent difference in behavior when a regex is specified as a literal string in source code compared with a regex that is read from a file or received from user input. For example, JavaScript’s string.replace() function does not support any of these escapes. But the JavaScript language does support escapes like \n, \x0A, and \u000A in string literals. So when developing an application in JavaScript, \n is only interpreted as a newline when you add the replacement text as a string literal to your source code. Then the JavaScript interpreter then translates \n and the string.replace() function sees an actual newline character. If your code reads the same replacement text from a file, then string.replace() function sees \n, which it treats as a literal backslash and a literal n.