Encodings: UTF‑8 / UTF‑16 / ASCII

1. ASCII — how it all started

ASCII is like the grandpa of the encoding world: it’s been around for ages, it knows neither emoji nor Cyrillic, but it still shows up at family gatherings. It’s important to understand ASCII not as “some old thing you can forget”, but as a baseline reference point: a lot of compatibility and many practical rules grew out of it.

ASCII (in its classic form) is a table of 128 characters (7 bits): Latin letters, digits, punctuation marks, and control characters like newline. So Hello, 123, A-Z, {}, + are OK. But Hello or 你好 are not—“not included”.

What happens if you try to “cram” Cyrillic into ASCII

In practice, when you encode a string as ASCII and the string contains a character that ASCII doesn’t know, it gets replaced with a placeholder character. Very often, it’s the question mark ?. This is not a “compiler error” and not “bad Kotlin”—it’s the honest reaction of the encoding: “I can’t do it, here’s a surrogate”.

fun main() {
    val original = "Hello"
    val bytes = original.toByteArray(Charsets.US_ASCII)

    val decoded = String(bytes, Charsets.US_ASCII)
    println(decoded) // ??????
}

The tricky part is that the program doesn’t crash. It just “quietly corrupts the data”. This is one of the most unpleasant classes of bugs: everything runs, but the result is wrong.

Why ASCII is still important

ASCII matters for two reasons. First, many protocols and formats (especially older ones) were historically designed around it. Second, UTF‑8 (more on it soon) is designed so that any pure ASCII text in UTF‑8 looks byte-for-byte the same as in ASCII. Because of that, ASCII often “hides” inside UTF‑8 as a special case.

2. UTF‑8 — the de facto standard

If ASCII is the grandpa, then UTF‑8 is that “universal language” most of the internet speaks today. It can encode all of Unicode (i.e., almost everything people have invented to write), while preserving the super-important property of compatibility with ASCII for plain text.

The main practical idea of UTF‑8: a variable-length encoding. One character can take 1, 2, 3, or 4 bytes. This means that “1 character = 1 byte” works only in a very narrow world of Latin letters and a couple of symbols.

UTF‑8 ASCII compatibility: why Hello is the same

Let’s verify it by hand: in ASCII and in UTF‑8, the bytes for English text will be identical. This is one reason why UTF‑8 is so convenient to use everywhere: old ASCII data doesn’t break.

fun main() {
    val s = "Hello"

    val ascii = s.toByteArray(Charsets.US_ASCII)
    val utf8 = s.toByteArray(Charsets.UTF_8)

    println(ascii.size)                // 5
    println(utf8.size)                 // 5
    println(ascii.contentEquals(utf8))  // true
}

So if your file contains only ASCII characters, you often won’t even notice the difference between “ASCII” and “UTF‑8” (which can make you relax—and that’s exactly why it hurts later when the first letter Zh shows up).

Why Hello in UTF‑8 is bigger than it seems

Cyrillic in UTF‑8 usually takes 2 bytes per letter. The word “Hello” has 6 letters—so that’s about 12 bytes. Let’s see:

fun main() {
    val ru = "Hello"

    val utf8 = ru.toByteArray(Charsets.UTF_8)
    println("utf8 bytes = ${utf8.size}") // utf8 bytes = 12
}

That’s normal. Those are not “extra bytes”; it’s the price of universality: UTF‑8 is compact for English text, but it honestly expands for other languages.

Emoji in UTF‑8: why 4 bytes is normal

Emoji are a classic example of characters that take 4 bytes in UTF‑8.

fun main() {
    val emoji = "🙂"
    val bytes = emoji.toByteArray(Charsets.UTF_8)

    println(bytes.size) // 4
}

If you ever have a task like “limit a message to N bytes” (for example, a network protocol, a field limit, a file size), you can already feel why a naive text.length doesn’t guarantee the right result at all.

3. UTF‑16 and the JVM reality

UTF‑16 is often perceived as a simple idea: “it’s where you have 2 bytes per character”. And in a everyday sense, that’s sometimes true… until the first emoji. And also until the moment you run into byte order and service stuff at the beginning of a file (but today we’ll only hint at that, without a deep dive).

The practical picture of UTF‑16 is this: it encodes Unicode using 16-bit code units. On the JVM, the Char type is exactly a 16-bit value. Kotlin on the JVM inherits Java’s model here: Char stores a 16-bit value (more precisely, a 16-bit UTF‑16 code unit).

Why String.length can “lie” for emoji

A very common cognitive shock: a string containing a single emoji can have length 2.

fun main() {
    val s = "🙂"
    println(s.length) // 2
}

This happens because in UTF‑16, some characters (outside the Basic Multilingual Plane) are encoded as a pair of 16-bit values. This is not about “memorizing tables”; it’s simply an important practical warning: length on the JVM is the number of Char, not “the number of visible characters”.

If that sounds like “I just want to read a file, not study Unicode philosophy”, then you’ve got it exactly right: our goal is not to go deep, but to stop making dangerous assumptions.

UTF‑16 size in bytes: why there are sometimes “extra” 2 bytes

UTF‑16 for Cyrillic usually gives 2 bytes per letter (so “Hello” ≈ 12 bytes), but in practice, when encoding as Charsets.UTF_16, additional service bytes may appear at the beginning. So it’s better to compare sizes carefully, with the understanding that UTF_16 in the API may include a byte-order marker.

Here’s a safe way to see that the size differs:

fun main() {
    val ru = "Hello"

    val utf8 = ru.toByteArray(Charsets.UTF_8)
    val utf16 = ru.toByteArray(Charsets.UTF_16)

    println("utf8=${utf8.size}")   // utf8=12
    println("utf16=${utf16.size}") // utf16=14
}

Why 14 and not 12? Because 2 service bytes were added at the beginning on top of the 12 bytes of data. Yes, that’s the “surprise” that later breaks reading the first CSV column, because the first header suddenly starts not with i, but with “an invisible thing + i”. The details of that “invisible guest” are better discussed separately so as not to mix everything into one mush.

How to see UTF‑16 bytes: printing hex

Printing a Byte directly isn’t very pleasant: it’s signed (from -128 to 127), and it’s easy to get confused in the console. For diagnostics, it’s convenient to print bytes as hex.

fun main() {
    val bytes = "A".toByteArray(Charsets.UTF_16)

    val hex = bytes.joinToString(" ") { b ->
        b.toUByte().toString(16).padStart(2, '0')
    }
    println(hex) // fe ff 00 41
}

Here we see: first fe ff (service bytes), then 00 41 (that’s the letter A). Even if you don’t yet know “what fe ff is”, you already see the main idea: in UTF‑16, the beginning of a byte sequence may contain not “text data”, but a service part.

4. Mini practice: an expense tracker and a text format

So that our knowledge doesn’t live separately from real life, let’s tie it to the application we’re building in the course. Suppose we have a console expense tracker that stores data line by line, for example like this:

100;food;Lunch
250;transport;Taxi

Parsing logic usually looks like “read a line → split by ; → parse numbers”. And this is where encodings start to affect things in an unexpected way: not split, but what is considered characters of the string at all.

Today we won’t touch file reading/writing with an explicit encoding (that’s a separate topic). But we can model the problem “in memory”: take a string, encode it as UTF‑16, and decode it as UTF‑8. That is classic mojibake.

fun main() {
    val header = "amount;category;comment"

    val bytesUtf16 = header.toByteArray(Charsets.UTF_16)
    val wrongText = String(bytesUtf16, Charsets.UTF_8)

    println(wrongText == header) // false
}

The key point: we don’t have to print wrongText (in the console it may look different), but we know for sure the contract is broken. We encoded by one set of rules and read by another—and got different text.

That’s why in real programs “the file reads fine for me but breaks for my colleague” very often boils down to one side silently using UTF‑8, while the other side silently uses UTF‑16 or the system encoding.

5. Practical rules for working with encodings

With encodings, it’s easy to start “doing magic”: switch UTF_8 to UTF_16, then to Windows-1251, then back, until it looks “pretty”. This works about as well as treating a headache by changing wallpapers: sometimes it helps, but you didn’t understand the cause.

Normal engineering discipline looks like this.

Comparing UTF‑8 / UTF‑16 / ASCII in practice

When people compare encodings, it’s easy to drift into Wikipedia and lose the point. We’ll do it simpler: compare the way a developer thinks when writing a console app and just wants the file to be read the same way everywhere.

First—a short table. Not for cramming, but for orientation.

Now let’s lock in the key practical takeaway that saves nerves: bytes depend on the encoding, which means file size, the “visibility” of characters, and parsing correctness all depend on it too.

Rules that help you avoid becoming a shaman

First, keep in mind that ASCII is not a universal encoding. If you see “??????” instead of Cyrillic, it almost always means the data was “packed” using an encoding that’s too limited (or someone decided the world consists of Latin letters).

Then remember that UTF‑8 is almost always the best choice for exchanging text externally: it’s compact for English, supports all of Unicode, and is compatible with ASCII where that matters (for example, headers, keys, structural characters). So if you have no special requirements, “agree on UTF‑8” is usually more reasonable than “leave it however it happens”.

And finally, treat UTF‑16 carefully: it’s neither bad nor good, it’s just different. Service bytes at the beginning often pop up, and on the JVM many things at the Char and length level behave like “the number of 16-bit chunks”, not “the number of visible characters”. That’s fine—just don’t write code that assumes otherwise.

If you collect this into one short “mantra without magic”, you get:

Text ≠ bytes.
An encoding is an agreement on translation rules.
Break the agreement — get mojibake.

6. Typical mistakes when comparing encodings

Mistake #1: thinking “1 character = 1 byte”.
This works only in the ASCII world and only for a limited set of characters. As soon as you hit Cyrillic, diacritics, Chinese ideographs, or emoji, the byte size starts to differ. If you build logic like “truncate a string to N bytes” using substring(0, n), you are almost guaranteed to get an incorrect result.

Mistake #2: assuming ASCII and UTF‑8 are the same thing.
They do match for many texts (for example, Hello), and that is misleading. In practice, they are identical only on the intersection of ASCII characters. As soon as Zh appears, ASCII turns it into ?, while UTF‑8 encodes it just fine.

Mistake #3: expecting String.length to answer the question “how many characters are in the text”.
On the JVM, length counts the number of Char, i.e., 16-bit code units. So one visual symbol can take two Char (a classic example is some emoji). That doesn’t make length useless, but it does make it dangerous to assume that length == the number of visible characters.

Mistake #4: comparing text sizes “by eye” and drawing conclusions about the encoding.
Sometimes a developer sees that a file is “bigger than expected” and decides: “so it’s garbage”. In reality, “bigger” may only mean the file is in a different encoding (for example, UTF‑16) or that it contains many non-ASCII characters. Size is a hint, not proof.

Mistake #5: trying to “fix mojibake” by replacing characters instead of fixing the encoding.
Replacements like text.replace("�", "") look like a quick fix, but usually it’s an attempt to wipe the consequences instead of the cause. The correct fix is almost always found at the boundary of “bytes ↔ string”: you need to decode the original bytes using the correct encoding, not patch up already broken text.