Four ways to count to four: PHP strings, iconv, mbstring and grapheme

Nothing like starting the day with a nice café, in a Parisian café, pun intended. Now, when you ask PHP how long the word "Café" is, you get four answers from four families of functions. Two of them agree, one says 5, and the fourth is waiting for you to add an emoji before it gets interesting. A croissant 🥐 for sure.

PHP is blessed with no more than four string toolboxes. Until recently, I was only aware of three of them, so I’ll start by a review of them all:

• Native string functions, such as strlen(), substr(), strtoupper()…, which count bytes. They’re fast, always available, and they have no idea what a character is. Did I say they count bytes?
• iconv, with iconv_strlen(), iconv_substr()… count code points. Code points are identifier for characters in an encoding, such as Unicode, but not only. They cover a huge number of characters, beyond the 256 range of bytes. iconv is a small extension with a narrow job: converting between encodings.
• mbstring, with such function as mb_strlen(), mb_substr()… also counts code points. It’s the big extension, with more than 60 functions. They overlap native functions, and deal with multi-bytes strings.
• grapheme functions from intl, with grapheme_strlen()…, count what a human would call a character. That solves the problem of characters build with different code points. This is a nuance, and it pays to know where it brings value to the code.

Let’s see where all these functions and extension agree, and where they don’t. Obviously, they will.

Round 1: how long is a string?

<?php
$nfc  = "Café";          // é as one code point (U+00E9)
$nfd  = "Cafe\u{0301}";  // é as two code points, e + combining acute accent
$fam  = "👨‍👩‍👧";            // family emoji (no relationship with the above)
$flag = "🇫🇷";            // a flag, obviously

//                 strlen  mb_strlen  iconv_strlen  grapheme_strlen
// $nfc               5        4           4              4
// $nfd               6        5           5              4
// $fam              18        5           5              1
// $flag              8        2           2              1
?>

A whole table of figures to count a string does require some explanations:

• strlen() counts bytes. That’s not a bug, it’s the job description. Use it for buffers, binary protocols and Content-Length.
• mb_strlen() and iconv_strlen() agree on everything: they both count Unicode code points.
• Only grapheme_strlen() sees what your users see. The accented é has two spellings in Unicode, called precomposed and decomposed, and only grapheme gives both the same length. The family emoji is five code points glued together with zero-width joiners. The flag is two “regional indicator” letters, F and R, which fonts merge into a flag.

If you validate a “max 20 characters” username field with mb_strlen(), a user with a family emoji in their nickname gets charged 5 characters for it. If you use strlen(), they get charged 18. That’s steep for a family.

Round 2: how to slice a string

Measuring is harmless. Cutting is where things get serious and break:

<?php
echo substr("Café", 0, 4);                  // "Caf\xC3": half an é, so you get �
echo mb_substr("Cafe\u{0301}", 0, 4);       // "Cafe": the accent was left behind
echo iconv_substr("Cafe\u{0301}", 0, 4);    // "Cafe": same
echo grapheme_substr("Cafe\u{0301}", 0, 4); // "Café"

echo mb_substr("👨‍👩‍👧", 0, 1);       // "👨": dad, alone
echo grapheme_substr("👨‍👩‍👧", 0, 1); // "👨‍👩‍👧": the whole family
?>

substr() cuts in the middle of a multibyte sequence and leaves invalid UTF-8 behind. Your database may reject it, or quietly save a lovely ?. mb_substr() never breaks a code point, but it can still separate a letter from its accent, or a dad from his family. Only grapheme_substr() keeps a grapheme together.

Negative offsets work everywhere. Three families agree on the result, and the fourth brings its own:

<?php
substr("Crème", -3);          // "\xA8me": the second half of è, then "me"
mb_substr("Crème", -3);       // "ème"
iconv_substr("Crème", -3);    // "ème"
grapheme_substr("Crème", -3); // "ème"
?>

Half a crème is still less than you ordered.

Round 3: how to search in a string

This is where the families are closest, and also where the surprises hide.

<?php
$s = "Déjà vu";
strpos($s, 'vu');          // 7 (bytes)
mb_strpos($s, 'vu');       // 5
iconv_strpos($s, 'vu');    // 5
grapheme_strpos($s, 'vu'); // 5
?>

Here’s the useful bit: for valid UTF-8, strpos() is correct for “is this in there?” A UTF-8 sequence can never match in the middle of another one, so str_contains(), str_starts_with() and str_ends_with() are safe. This is lucky, but it works.

You only need the mb_ version when you want the position as a character offset, for example to hand it to mb_substr(). Mixing byte positions from strpos() with mb_substr() is a classic bug.

Case-insensitive search is a different story:

<?php
stripos("CAFÉ", "é");          // false: É and é are different bytes, no recognized as upper and lower case
mb_stripos("CAFÉ", "é");       // 3
grapheme_stripos("CAFÉ", "é"); // 3
// iconv_stripos()?  The function doesn't exist.
?>

And grapheme has opinions about what counts as a match:

<?php
$nfd = "Cafe\u{0301}";
strpos($nfd, 'e');          // 3
mb_strpos($nfd, 'e');       // 3
grapheme_strpos($nfd, 'e'); // false: that 'e' belongs to 'é'

mb_strpos($nfd, "é");       // false: precomposed é isn't in there, byte-wise
grapheme_strpos($nfd, "é"); // 3: grapheme finds it anyway
?>

Depending on your use case, that’s either exactly right or a nasty surprise. Search engines and word filters love it. Code that searches with grapheme_strpos() and then cuts with substr() does not.

Some more edge cases

Out-of-range offsets are one area where all four agree: they throw the same ValueError.

<?php
strpos("abc", "a", 10);
// ValueError: strpos(): Argument #3 ($offset) must be contained in argument #1 ($haystack)
// same for mb_strpos(), iconv_strpos() and grapheme_strpos()
?>

Empty needles are where they split:

<?php
strpos("abc", "");          // 0
mb_strpos("abc", "");       // 0
grapheme_strpos("abc", ""); // 0
iconv_strpos("abc", "");    // false
?>

iconv never got the PHP 8.0 memo about empty needles. It’s been busy converting charsets.

Round 4: upper, lower and the case of longer uppercase strings

<?php
strtoupper("café");       // "CAFé"
mb_strtoupper("café");    // "CAFÉ"
mb_strtoupper("straße");  // "STRASSE": full case mapping
mb_convert_case("straße", MB_CASE_UPPER_SIMPLE); // "STRAßE"

ucfirst("élan");          // "élan"
mb_ucfirst("élan");       // "Élan" (PHP 8.4+)

ucwords("l'été à paris");                        // "L'été à Paris"
mb_convert_case("l'été à paris", MB_CASE_TITLE); // "L'été À Paris"
?>

Since PHP 8.2, strtoupper() and friends ignore the locale configuration of the system and only handle ASCII. That way, th feature is more predictable, and it’s also wrong for anything past the letter z. mbstring handles Unicode case mapping, including the German ß, which turns into two letters when uppercased. So that universal truth that “uppercased string has the same length” assumption just failed an exam.

iconv has no case conversion functions at all.

Grapheme also has no case conversion, but that is handled by the rest of the intl extension, which is the host of Grapheme: the IntlCharhas the methods you need. Grapheme’s case-insensitive search does case folding internally, so it won’t give you the result.

Round 5: how to deal with invalid strings

Now let’s feed them invalid strings. By that, we mean strings containing invalid UTF-8 since PHP string are always valid as the byte level. Yet, invalid UTF-8 is the the way of real life:

<?php
$bad = "abc\xFFdef";

strlen($bad);          // 7
mb_strlen($bad);       // 7: the bad byte counts as one character
iconv_strlen($bad);    // false + Notice: Detected an illegal character in input string
grapheme_strlen($bad); // null (!), intl_get_error_message() says U_INVALID_CHAR_FOUND
?>

Four functions, four attitudes:

• native doesn’t care, as expected. It’s valid.
• mbstring keeps going,
• iconv complains,
• grapheme returns null, which isn’t even in the traditional int|false return value.

So if (grapheme_strlen($s) === false) will never catch that error. Validate first with mb_check_encoding($s, 'UTF-8'), or clean up with mb_scrub(), which gives "abc?def".

Round 6: all together now, splitting, padding, trimming, reversing

<?php
str_split("été");              // ["\xC3", "\xA9", "t", "\xC3", "\xA9"]: confetti
mb_str_split("été");           // ["é", "t", "é"]
grapheme_str_split("👨‍👩‍👧🇫🇷"); // ["👨‍👩‍👧", "🇫🇷"] (PHP 8.4+)

str_pad("été", 6, "*");        // "été*": it thinks été is already 5 long
mb_str_pad("été", 6, "*");     // "été***" (PHP 8.3+)

trim("\u{00A0}été");           // untouched: the non-breaking space isn't ASCII
mb_trim("\u{00A0}été\u{3000}"); // "été" (PHP 8.4+)

strrev("été");                 // broken UTF-8
// mb_strrev()? Doesn't exist. Neither does grapheme_strrev().
?>

mbstring has been catching up quickly: mb_str_pad() in 8.3, then mb_trim(), mb_ltrim(), mb_rtrim(), mb_ucfirst() and mb_lcfirst() in 8.4. There’s still no mb_strrev() though, but it may come some day. The usual workaround is implode('', array_reverse(grapheme_str_split($s))), and yes, it should be grapheme. Reversing by code points moves accents onto the wrong letters, which is a creative way to write French.

Round 7: converting strings

iconv has 10 only functions, and the most important one is the eponymous one: iconv(). This is where it earns its place.

<?php
iconv('UTF-8', 'ASCII//TRANSLIT', 'Crème brûlée'); // "Creme brulee"
iconv('UTF-8', 'ASCII//IGNORE',   'Crème brûlée'); // "Crme brle": not even a desert
iconv('UTF-8', 'ASCII',           'Crème brûlée'); // false + notice

mb_convert_encoding('Crème brûlée', 'ASCII', 'UTF-8'); // "Cr?me br?l?e"
?>

//TRANSLIT is great for slugs, but there’s a catch: the result depends on the underlying C library. The output above is from glibc, so check ICONV_IMPL. On Alpine’s musl, or on older libiconv builds, you may get ? or false instead. Your Docker image just became part of your business logic. If you need a slug that’s reproducible everywhere, intl’s Transliterator::transliterate('Any-Latin; Latin-ASCII', $s) is the portable choice.

For plain conversions like UTF-8 <=> ISO-8859-1, both work:

<?php
bin2hex(iconv('UTF-8', 'ISO-8859-1', 'é'));               // "e9"
bin2hex(mb_convert_encoding('é', 'ISO-8859-1', 'UTF-8')); // "e9"
?>

mbstring also has encoding detection, with mb_detect_encoding(), to be taken with a large grain of salt, HTML entities, Japanese kana conversions with mb_convert_kana(), and display width:

<?php
mb_strwidth("日本");                             // 4: CJK characters are double-width
mb_strimwidth("Hello 日本語の世界", 0, 10, "…"); // "Hello 日…"
?>

This is handy for aligning console tables, and nobody else offers it.

What the string families have in common

Before the scoreboard, here’s what’s shared, because it’s more than you’d expect:

• Same naming pattern, strlen -> mb_strlen / iconv_strlen / grapheme_strlen. The same goes for substr, strpos, strrpos. Swapping one family for another is mostly a search-and-replace, which makes it tempting to do without thinking. Just remember some behavior and return values may change.
• Same signatures and conventions. Negative offsets, false when nothing is found, and ValueError on out-of-range offsets since PHP 8.0.
• Same answer on simple text. On ASCII, all four agree. That’s exactly why the bugs survive testing: your test suite is written in English.
• Configurable encoding for mbstring and iconv: mb_strlen($s, 'ISO-8859-1'), iconv_strlen($s, 'ISO-8859-1'). Both default to default_charset, which is UTF-8. Grapheme is UTF-8 only. By the way, mb_strlen($s, '8bit') is a well-known alias for strlen().
• None of them normalize. "Café" === "Cafe\u{0301}" is false, whichever family you pick. Use Normalizer::normalize() from intl before comparing, hashing or storing unique keys.

The scoreboard

I just could not resist using emojis in an article about unicode and string manipulation. Sorry, not sorry.

So, which to use in what situation?

• Bytes, for storage, protocols, hashing, str_contains() on valid UTF-8: native functions. They’re not wrong, just literal. And fast.
• Text processing for case, trimming, padding, code-point offsets: mbstring. It’s the most popular for multilingual code in PHP codes, so it is safe. If you want to prepare for the future, consider moving to intl and grapheme functions.
• Encoding conversion and quick transliteration: iconv(), as long as you pay attention to which libc you’re running on. For the iconv_str* functions, there’s nothing mbstring doesn’t also do, with fewer notices.
• Anything a human will read, count or cut: truncating titles, length limits in forms, cursor positions, reversing: grapheme. It’s slower and returns odd values on bad input, but it’s the only one that knows a family is one thing.

And the golden rule: never mix units. A position from strpos() fed to mb_substr(), or from mb_strpos() fed to grapheme_substr(), will work perfectly until the first customer called Zoë or 李.

Happy coding, and may all your strings be exactly as long as they look! May be I’ll take a second 🥐 today…