Encoding: why “garbled text” appears

1. Introduction

If you have ever opened a file in Notepad and thought “well it’s text, it has letters,” you’re in good company: almost everyone thinks so until their first run-in with encodings. In reality, the operating system and the file system do not store “letters,” “strings,” or “paragraphs.” They store numbers from 0 to 255 (bytes), just in a very long sequence. And “letters” appear later—when some program takes those bytes and decides how to turn them into characters.

Imagine bytes as musical notes written as digits, and an encoding as the rule “which digit corresponds to which note” (and sometimes—how many digits are needed for one note). If you play music interpreting the digits with the wrong table, the result will be at least weird. Text is the same: the bytes are the same, but the interpretation rules are different—and instead of “Hello” you see “Привет” or “??????”.

It is very important to fix the boundary:

ByteArray

String

In Kotlin (on the JVM) we can “honestly” see the bytes of a file like this:

import java.io.File

fun main() {
    val bytes = File("data/note.txt").readBytes()
    println(bytes.size) // e.g.: 128
}

So far there is no text—only a ByteArray.

2. Encoding: the rule “bytes ↔ text”

In this section we will carefully introduce a new term, because without it it will be hard to continue the discussion. An encoding is not a “file format” and not a “language,” but a conversion rule between bytes and characters. And it is a two-way rule: how to turn bytes into text (decoding) and how to turn text into bytes (encoding). In other words, an encoding is part of the contract: “I wrote the bytes this way; you read them this way.”

It helps to think of an encoding as “glasses.” Bytes are the same object of reality. The encoding is the lens. Put on the right glasses—you see normal text. Put on the wrong ones—you see the same bytes, but as a meaningless set of symbols. And that is exactly why “garbled text” often looks stable: it is not random noise, but a very specific interpretation.

At the API level in Kotlin/JVM, an encoding is represented by the type Charset. We will most often use ready-made values from Charsets, for example Charsets.UTF_8. Today we will not dive into what encodings exist and how they differ—what matters is the principle: the encoding must be consistent between writing and reading.

To get a hands-on feel for the idea, let’s start simple: string → bytes.

fun main() {
    val text = "Hello"
    val bytes = text.toByteArray(Charsets.UTF_8)
    println(bytes.size) // 5 (for UTF-8, English letters take 1 byte)
}

Here we already made an important step: we said explicitly, “convert the string to bytes using UTF-8.”

3. Encoding and decoding: two directions of the same path

At this point it is easy to get confused by the terminology, so let’s lock it in at a practical level. When you have a ByteArray and you want to get a String, you do decoding. When you have a String and you want to write bytes to a file, you do encoding. These are mirror operations, and they must use the same encoding if you want to get the original text back.

It is very convenient to keep this in mind as a small pipeline. The diagram below explains almost 80% of mysteries like “why did the text break?”

flowchart TD
    A[String: 'Hello'] -->|"encode: toByteArray(UTF-8)"| B[ByteArray]
    B -->|write to file| C[(File on disk)]
    C -->|"readBytes()"| D[ByteArray]
    D -->|"decode: String(bytes, UTF-8)"| E[String: 'Hello']

In Kotlin both operations look pretty straightforward:

• Encoding: text.toByteArray(charset)
• Decoding: String(bytes, charset)

Mini example “there and back”:

fun main() {
    val original = "Hello"
    val bytes = original.toByteArray(Charsets.UTF_8)
    val decoded = String(bytes, Charsets.UTF_8)

    println(decoded) // Hello
}

As long as the encoding matches—everything is fine. “Garbled text” appears when the encodings do not match.

4. Diagnostics and “garbled text” in real code

How to “see the bytes” so you stop guessing

When you first study encodings, a typical situation happens: “I opened a file, I see nonsense, I tried another encoding—I see different nonsense.” It quickly turns into shamanism. To avoid guessing, it helps to be able to print bytes in a readable form, for example in hex (hexadecimal). This is not magic and not “for hackers”—it is simply a convenient way to see what is actually stored in the file.

Let’s add a small utility to our learning project: convert a ByteArray into a string of hex values. Here we use simple formatting and joinToString().

fun toHex(bytes: ByteArray): String {
    return bytes.joinToString(" ") { b -> "%02X".format(b) }
}

fun main() {
    val bytes = byteArrayOf(72, 101, 108, 108, 111)
    println(toHex(bytes)) // 48 65 6C 6C 6F
}

Now we can diagnose: “the file actually contains bytes 48 65 6C 6C 6F.” That is no longer feelings—it is facts.

By the way, the same bytes can be decoded immediately as UTF-8 (or ASCII—for English text they will match):

fun main() {
    val bytes = byteArrayOf(72, 101, 108, 108, 111)
    val text = String(bytes, Charsets.UTF_8)

    println(text) // Hello
}

How “garbled text” appears

Now—the central focus of the lecture. “Garbled text” is almost always the situation “the bytes are correct, but we are reading them with the wrong rules.” That is, the problem is not that “the file got corrupted,” but that we performed the wrong decoding (or earlier wrote it using the wrong encoding). The simplest way to see this is to encode text one way and decode it another way.

Let’s take an English word, encode it in UTF-8, and then try to read those bytes as ASCII. ASCII is very limited (basically Latin letters and control characters), so everything outside it will be turned into “replacement” characters (often ?).

fun main() {
    val original = "Hello"
    val bytes = original.toByteArray(Charsets.UTF_8)

    val wrong = String(bytes, Charsets.US_ASCII)
    println(wrong) // Hello  (often like this; depends on replacement)
}

Important: the bytes did not change. The interpretation rule changed. That is why the output is not a “broken file,” but a logical result of “ASCII cannot represent the needed characters.”

You can do another illustrative trick: for the word "Hello", the bytes in ASCII and UTF-8 are the same. That is why sometimes it feels like “encoding doesn’t matter”: for English text, many encodings “accidentally work.”

fun main() {
    val s = "Hello"
    val a = s.toByteArray(Charsets.US_ASCII)
    val u = s.toByteArray(Charsets.UTF_8)

    println(a.contentEquals(u)) // true
}

But for non-ASCII text you will not get such a match, and that is normal.

Example: “BudgetBuddy” and the I/O boundary

Now we will take a small but very practical step toward a “grown-up” application. Imagine we have a console project BudgetBuddy (expense tracking) that stores data in a text file "data/expenses.txt". We already know how to read text via readText(), but today we want to understand what happens at the “file → string” boundary so we can debug cases where the file came from another system or editor.

The idea is simple: we will make a function that reads a file as bytes and turns it into a string with explicit decoding. Today we will fix only the principle itself (decode explicitly)—the choice of a specific encoding and the discipline of “don’t rely on the system default” will be expanded in the next lectures of the day.

import java.io.File

fun readTextUtf8(file: File): String {
    val bytes = file.readBytes()
    return String(bytes, Charsets.UTF_8)
}

fun main() {
    val text = readTextUtf8(File("data/expenses.txt"))
    println(text.length) // e.g.: 57
}

Yes, this looks a bit “longer” than readText(). But now we have a point where we can:

• print the first bytes,
• make sure the file is actually not empty,
• check that we are decoding using exactly the rule we expect.

For example, if we suspect the file is being read “wrong,” we can print the first few bytes:

import java.io.File

fun main() {
    val bytes = File("data/expenses.txt").readBytes()
    val preview = bytes.take(12).toByteArray()

    println(toHex(preview)) // e.g.: 50 61 79 6D 65 6E 74 3A 20 ...
}

Note: we are not “fixing” anything here. We are doing what programmers do best—we stop guessing and start measuring.

Why “it looks fine in the editor” is not a guarantee

This section is needed to remove one popular illusion. A student often says: “But I opened the file in the editor, and it looks fine. Why does the program show garbled text?” This is a great question because it points to the key idea: the editor also chooses an encoding. Sometimes automatically, sometimes by project settings, sometimes by “guessing.” So you see it “fine” not because the file is “magically correct,” but because the editor guessed correctly or you configured it correctly.

The program, on the other hand, decodes the bytes the way you told it (or whatever happens “by default”). Therefore, the situation “the editor shows it fine, but the program doesn’t” means: the editor and the program chose different decoding rules. And that brings us back to the main thesis of the lecture: text is bytes + rules.

Once you accept this model, many oddities stop being odd. You stop thinking “the file is cursed” and start thinking “which charset was used when writing, and which one when reading.” And that is a problem that has a solution.

5. Common mistakes

Mistake #1: assuming that String “lives in the file” directly as a String.
This is one of the most persistent misconceptions. In Kotlin, String is an object in the program’s memory. In a file—only bytes. If you keep that in mind, you will automatically look for the problem at the “bytes ↔ text” boundary instead of trying to fix “wrong characters” with weird replace().

Mistake #2: trying to “fix garbled text” by replacing characters instead of decoding correctly.
Sometimes you want to do something like text.replace("Ð", "P"). This is almost always a path to even more chaos: you are treating the symptom, not the cause. The correct repair point is choosing the right encoding during decoding.

Mistake #3: thinking “1 character = 1 byte.”
This is a common trap after the first lessons on ByteArray. For ASCII it often “seems” true, but for Unicode text the size in bytes can be larger. If you start slicing text “by bytes” while expecting “by characters,” you can easily corrupt data.

Mistake #4: debugging “by eye” without printing the bytes.
When you have a problem at the byte level, looking only at println(text) is like trying to understand a bug in arithmetic without printing the numbers. Simple diagnostics via a hex representation of the first bytes often saves hours.

Mistake #5: mixing levels: reading a file, parsing, and validating in one place.
When everything is done in one big chunk of code, you do not understand where exactly things went wrong: the file was not read, the bytes are wrong, decoding is wrong, or the parser complains. It is better to have a separate step “get bytes,” a separate step “decode into a string,” and only then—business logic.