Strings and Characters
**********************

This chapter describes the APIs for strings and characters, provided by Gnulib.

Strings
=======

Several possible representations exist for the representation of strings in memory of a running C program.

The C string representation
---------------------------

The classical representation of a string in C is a sequence of characters, where each character takes up one or more bytes, followed by a terminating NUL byte.  This representation is used for strings that are passed by the operating system (in the argv argument of main, for example) and for strings that are passed to the operating system (in system calls such as open).  The C type to hold such strings is char * or, in places where the string shall not be modified, const char *.  There are many C library functions, standardized by ISO C and POSIX, that assume this representation of strings.

A character encoding, or encoding for short, describes how the elements of a character set are represented as a sequence of bytes.  For example, in the ASCII encoding, the UNDERSCORE character is represented by a single byte, with value 0x5F.  As another example, the COPYRIGHT SIGN character is represented:

  * in the ISO-8859-1 encoding, by the single byte 0xA9,

  * in the UTF-8 encoding, by the two bytes 0xC2 0xA9,

  * in the GB18030 encoding, by the four bytes 0x81 0x30 0x84 0x38.

Note: The char type may be signed or unsigned, depending on the platform.  When we talk about the "byte 0xA9" we actually mean the char object whose value is (char) 0xA9; we omit the cast to char in this documentation, for brevity.

In POSIX, the character encoding is determined by the locale.  The locale is some environmental attribute that the user can choose.

Depending on the encoding, in general, every character is represented by one or more bytes (up to 4 bytes in practice -- but use MB_LEN_MAX instead of the number 4 in the code). When every character is represented by only 1 byte, we speak of an ``unibyte locale'', otherwise of a ``multibyte locale''.

It is important to realize that the majority of Unix installations nowadays use UTF-8 as locale encoding; therefore, the majority of users are using multibyte locales.

Three important facts to remember are:

A char is a byte, not a character.

As a consequence:

  * The ctype.h API, that was designed only with unibyte encodings in mind, is useless nowadays for general text processing; it does not work in multibyte locales.

  * The strlen function does not return the number of characters in a string.  Nor does it return the number of screen columns occupied by a string after it is output.  It merely returns the number of bytes occupied by a string.

  * Truncating a string, for example, with strncpy, can have the effect of truncating it in the middle of a multibyte character.  Such a string will, when output, have a garbled character at its end, often represented by a hollow box.

Multibyte does not imply UTF-8 encoding.

While UTF-8 is the most common multibyte encoding, GB18030 is also a supported locale encoding on GNU systems (mostly because it is a Chinese government standard, last revised in 2022).

Searching for a character in a string is not the same as searching for a byte in the string.

Take the above example of COPYRIGHT SIGN in the GB18030 encoding: A byte search will find the bytes '0' and '8' in this string.  But a search for the character "0" or "8" in the string "©" must, of course, report ``not found''.

As a consequence:

  * strchr and strrchr do not work with multibyte strings if the locale encoding is GB18030 and the character to be searched is a digit.

  * strstr does not work with multibyte strings if the locale encoding is different from UTF-8.

  * strcspn, strpbrk, strspn cannot work correctly in multibyte locales: they assume the second argument is a list of single-byte characters.  Even in this simple case, they do not work with multibyte strings if the locale encoding is GB18030 and one of the characters to be searched is a digit.

  * strsep and strtok_r do not work with multibyte strings unless all of the delimiter characters are ASCII characters < 0x30.

  * The strcasecmp, strncasecmp, and strcasestr functions do not work with multibyte strings.

Workarounds can be found in Gnulib, in the form of mbs* API functions:

  * Gnulib has functions mbslen and mbswidth that can be used instead of strlen when the number of characters or the number of screen columns of a string is requested.

  * Gnulib has functions mbschr and mbsrrchr that are like strchr and strrchr, but work in multibyte locales.

  * Gnulib has a function mbsstr that is like strstr, but works in multibyte locales.

  * Gnulib has functions mbscspn, mbspbrk, mbsspn that are like strcspn, strpbrk, strspn, but work in multibyte locales.

  * Gnulib has functions mbssep and mbstok_r that are like strsep and strtok_r but work in multibyte locales.

  * Gnulib has functions mbscasecmp, mbsncasecmp, mbspcasecmp, and mbscasestr that are like strcasecmp, strncasecmp, and strcasestr, but work in multibyte locales.  Still, the function ulc_casecmp is preferable to these functions.

A C string can contain encoding errors.

Not every NUL-terminated byte sequence represents a valid multibyte string.  Byte sequences can contain encoding errors, that is, bytes or byte sequences that are invalid and do not represent characters.

String functions like mbscasecmp and strcoll whose behavior depends on encoding have unspecified behavior on strings containing encoding errors, unless the behavior is specifically documented.  If an application needs a particular behavior on these strings it can iterate through them itself, as described in the next subsection.

Iterating through strings
-------------------------

For complex string processing, string functions may not be enough, and you need to iterate through a string while processing each (possibly multibyte) character or encoding error in turn.  Gnulib has several modules for iterating forward through a string in this way.  Backward iteration, that is, from the string's end to start, is not provided, as it is too hairy in general.

  * The mbiter module iterates through a string whose length is already known.  The string can contain NULs and encoding errors.

  * The mbiterf module is like mbiter except it is more complex and typically faster.

  * The mbuiter module iterates through a C string whose length is not a-priori known.  The string can contain encoding errors and is terminated by the first NUL.

  * The mbuiterf module is like mbuiter except it is more complex and typically faster.

  * The mcel module is simpler than mbiter and mbuiter and can be faster than even mbiterf and mbuiterf. It can iterate through either strings whose length is known, or C strings, or strings terminated by other ASCII characters < 0x30.

  * The mcel-prefer module is like mcel except that it causes some other modules to be based on mcel instead of on the mbiter family.

The choice of modules depends on the application's needs.  The mbiter module family is more suitable for applications that treat some sequences of two or more bytes as a single encoding error, and for applications that need to support obsolescent encodings on non-GNU platforms, such as CP864, EBCDIC, Johab, and Shift JIS. In this module family, mbuiter and mbuiterf are more suitable than mbiter and mbiterf when arguments are C strings, lengths are not already known, and it is highly likely that only the first few multibyte characters need to be inspected.

The mcel module is simpler and can be faster than the mbiter family, and is more suitable for applications that do not need the mbiter family's special features.

The mcel-prefer module is like mcel except that it also causes some other modules, such as mbscasecmp, to use mcel rather than the mbiter family.  This can be simpler and faster.  However, it does not support the obsolescent encodings, and it may behave differently on data containing encoding errors where behavior is unspecified or undefined, because in mcel each encoding error is a single byte whereas in the mbiter family a single encoding error can contain two or more bytes.

If a package uses mcel-prefer, it may also want to give gnulib-tool one or more of the options --avoid=mbiter, --avoid=mbiterf, --avoid=mbuiter and --avoid=mbuiterf, to avoid packaging modules that are not needed.

Strings with NUL characters
---------------------------

The GNU Coding Standards, section Writing Robust Programs (standards), specifies:

Utilities reading files should not drop NUL characters, or any other nonprinting characters.

When it is a requirement to store NUL characters in strings, a variant of the C strings is needed.  Gnulib offers a ``string descriptor'' type for this purpose.  See Handling strings with NUL characters.

All remarks regarding encodings and multibyte characters in the previous section apply to string descriptors as well.

     [file c-locale.texi not included]

Comparison of string APIs
-------------------------

This table summarizes the API functions available for strings, in POSIX and in Gnulib.

unibyte strings only
assume C locale
multibyte strings
multibyte strings with NULs
wide character strings
32-bit wide character strings

strlen
strlen
mbslen
sd_length
wcslen
u32_strlen

strnlen
strnlen
mbsnlen
--
wcsnlen
u32_strnlen, u32_mbsnlen

strcmp
strcmp
strcmp
sd_cmp
wcscmp
u32_strcmp

strncmp
strncmp
strncmp
--
wcsncmp
u32_strncmp

strcasecmp
strcasecmp
mbscasecmp
--
wcscasecmp
u32_casecmp

strncasecmp
strncasecmp
mbsncasecmp, mbspcasecmp
--
wcsncasecmp
u32_casecmp

strcoll
strcmp
strcoll
--
wcscoll
u32_strcoll

strxfrm
--
strxfrm
--
wcsxfrm
--

strchr
strchr
mbschr
sd_index
wcschr
u32_strchr

strrchr
strrchr
mbsrchr
sd_last_index
wcsrchr
u32_strrchr

strstr
strstr
mbsstr
sd_contains
wcsstr
u32_strstr

strcasestr
strcasestr
mbscasestr
--
--
--

str_
 startswith

str_
 startswith

mbs_
 startswith

sd_
 startswith
--

u32_
 startswith

str_endswith
str_endswith
mbs_endswith
sd_endswith
--
u32_endswith

strspn
strspn
mbsspn
--
wcsspn
u32_strspn

strcspn
strcspn
mbscspn
--
wcscspn
u32_strcspn

strpbrk
strpbrk
mbspbrk
--
wcspbrk
u32_strpbrk

strtok_r
strtok_r
mbstok_r
--
wcstok
u32_strtok

strsep
strsep
mbssep
--
--
--

strcpy
strcpy
strcpy
sd_copy
wcscpy
u32_strcpy

stpcpy
stpcpy
stpcpy
--
wcpcpy
u32_stpcpy

strncpy
strncpy
strncpy
--
wcsncpy
u32_strncpy

stpncpy
stpncpy
stpncpy
--
wcpncpy
u32_stpncpy

strcat
strcat
strcat
sd_concat
wcscat
u32_strcat

strncat
strncat
strncat
--
wcsncat
u32_strncat

free
free
free
sd_free
free
free

strdup
strdup
strdup
sd_copy
wcsdup
u32_strdup

strndup
strndup
strndup
--
--
--

mbswidth
mbswidth
mbswidth
--
wcswidth
c32swidth, u32_strwidth

strtol
strtol
strtol
--
--
--

strtoul
strtoul
strtoul
--
--
--

strtoll
strtoll
strtoll
--
--
--

strtoull
strtoull
strtoull
--
--
--

strtoimax
strtoimax
strtoimax
--
wcstoimax
--

strtoumax
strtoumax
strtoumax
--
wcstoumax
--

strtof
--
strtof
--
--
--

strtod
c_strtod
strtod
--
--
--

strtold
c_strtold
strtold
--
--
--

strfromf
--
strfromf
--
--
--

strfromd
--
strfromd
--
--
--

strfroml
--
strfroml
--
--
--

--
--
--
--
mbstowcs
mbstoc32s

--
--
--
--
mbsrtowcs
mbsrtoc32s

--
--
--
--
mbsnrtowcs
mbsnrtoc32s

--
--
--
--
wcstombs
c32stombs

--
--
--
--
wcsrtombs
c32srtombs

--
--
--
--
wcsnrtombs
c32snrtombs

Characters
==========

A character is the elementary unit that strings are made of.

What is a character?  ``A character is an element of a character set'' is sort of a circular definition, but it highlights the fact that it is not merely a number.  Although many characters are visually represented by a single glyph, there are characters that, for example, have a different glyph when used at the end of a word than when used inside a word.  A character is also not the minimal rendered text processing unit; that is a grapheme cluster and in general consists of one or more characters.  If you want to know more about the concept of character and various concepts associated with characters, refer to the Unicode standard.

For the representation in memory of a character, various types have been in use, and some of them were failures: char and wchar_t were invented for this purpose, but are not the right types. char32_t is the right type (successor of wchar_t); and mbchar_t (defined by Gnulib) is an alternative for specific kinds of processing.

The char type
-------------

The char type is in the C language since the beginning in the 1970ies, but -- due to its limitation of 256 possible values -- is no longer the adequate type for storing a character.

Technically, it is still adequate in unibyte locales.  But since most locales nowadays are multibyte locales, it makes no sense to write a program that runs only in unibyte locales.

ISO C and POSIX standardized an API for characters of type char, in <ctype.h>.  This API is nowadays useless and obsolete, when it comes to general text processing.

The important lessons to remember are:

A char is just the elementary storage unit for a string, not a character.

Never use <ctype.h>!

The wchar_t type
----------------

The ISO C and POSIX standard creators made an attempt to overcome the dead end regarding the char type.  They introduced

  * a type wchar_t, designed to encapsulate a character,

  * a ``wide string'' type wchar_t *, with some API functions declared in wchar.h, and

  * functions declared in wctype.h that were meant to supplant the ones in ctype.h.

Unfortunately, this API and its implementation has numerous problems:

  * On Windows platforms and on AIX in 32-bit mode, wchar_t is a 16-bit type.  This means that it can never accommodate an entire Unicode character.  Either the wchar_t * strings are limited to characters in UCS-2 (the ``Basic Multilingual Plane'' of Unicode), or -- if wchar_t * strings are encoded in UTF-16 -- a wchar_t represents only half of a character in the worst case, making the wctype.h functions pointless.

  * On Solaris and FreeBSD, the wchar_t encoding is locale dependent and undocumented.  This means, if you want to know any property of a wchar_t character, other than the properties defined by wctype.h -- such as whether it's a dash, currency symbol, paragraph separator, or similar --, you have to convert it to char * encoding first, by use of the function wctomb.

  * When you read a stream of wide characters, through the functions fgetwc and fgetws, and when the input stream/file is not in the expected encoding, you have no way to determine the invalid byte sequence and do some corrective action.  If you use these functions, your program becomes ``garbage in - more garbage out'' or ``garbage in - abort''.

As a consequence, it is better to use multibyte strings.  Such multibyte strings can bypass limitations of the wchar_t type, if you use functions defined in Gnulib and GNU libunistring for text processing. They can also faithfully transport malformed characters that were present in the input, without requiring the program to produce garbage or abort.

The char32_t type
-----------------

The ISO C and POSIX standard creators then introduced the char32_t type.  In ISO C 11, it was conceptually a ``32-bit wide character'' type.  In ISO C 23, its semantics has been further specified: A char32_t value is a Unicode code point.

Thus, the char32_t type is not affected the problems that plague the wchar_t type.

The char32_t type and its API are defined in the <uchar.h> header file.

ISO C and POSIX specify only the basic functions for the char32_t type, namely conversion of a single character (mbrtoc32 and c32rtomb).  For convenience, Gnulib adds API for classification and case conversion of characters.

GNU libunistring can also be used on char32_t values.  Since char32_t is the same as uint32_t, all u32_* functions of GNU libunistring are applicable to arrays of char32_t values.

On glibc systems, use of the 32-bit wide strings (char32_t[]) is exactly as efficient as the use of the older wide strings (wchar_t[]).  This is possible because on glibc, wchar_t values already always were 32-bit and Unicode code points. mbrtoc32 is just an alias of mbrtowc.  The Gnulib *c32* functions are optimized so that on glibc systems they immediately redirect to the corresponding *wc* functions.

Gnulib implements the ISO C 23 semantics of char32_t when you import the uchar-h-c23 module.  Without this module, it implements only the ISO C 11 semantics; the effect is that on some platforms (macOS, FreeBSD, NetBSD, Solaris) a char32_t value is the same as a wchar_t value, not a Unicode code point.  Thus, when you want to pass char32_t values to GNU libunistring or to some Unicode centric Gnulib functions, you need the uchar-h-c23 module in order to do so without portability problems.

The mbchar_t type
-----------------

Gnulib defines an alternate way to encode a multibyte character: mbchar_t.  Its main feature is the ability to process a string or stream with some malformed characters without reporting an error.

The type mbchar_t, defined in "mbchar.h", holds a character in both the multibyte and the 32-bit wide character representation.  In case of a malformed character only the multibyte representation is used.

Reading multibyte strings
.........................

If you want to process (possibly multibyte) characters while reading them from a FILE * stream, without reading them into a string first, the mbfile module is made for this purpose.

Comparison of character APIs
----------------------------

This table summarizes the API functions available for characters, in POSIX and in Gnulib.

unibyte character
assume C locale
wide character
32-bit wide character
mbchar_t character

== '\0'
== '\0'
== L'\0'
== 0
mb_isnul

==
==
==
==
mb_equal

isalnum
c_isalnum
iswalnum
c32isalnum
mb_isalnum

isalpha
c_isalpha
iswalpha
c32isalpha
mb_isalpha

isblank
c_isblank
iswblank
c32isblank
mb_isblank

iscntrl
c_iscntrl
iswcntrl
c32iscntrl
mb_iscntrl

isdigit
c_isdigit
iswdigit
c32isdigit
mb_isdigit

isgraph
c_isgraph
iswgraph
c32isgraph
mb_isgraph

islower
c_islower
iswlower
c32islower
mb_islower

isprint
c_isprint
iswprint
c32isprint
mb_isprint

ispunct
c_ispunct
iswpunct
c32ispunct
mb_ispunct

isspace
c_isspace
iswspace
c32isspace
mb_isspace

isupper
c_isupper
iswupper
c32isupper
mb_isupper

isxdigit
c_isxdigit
iswxdigit
c32isxdigit
mb_isxdigit

--
--
wctype
c32_get_type_test
--

--
--
iswctype
c32_apply_type_test
--

tolower
c_tolower
towlower
c32tolower
--

toupper
c_toupper
towupper
c32toupper
--

--
--
wctrans
c32_get_mapping
--

--
--
towctrans
c32_apply_mapping
--

--
--
wcwidth
c32width
mb_width