Cleaning “dirty text”

1. What “dirty text” is

When you read a file and get a String, it’s very tempting to think: “That’s it, it’s already text; from here on it’s just split, trim, comparisons.” But the outside world loves surprises: the file may have come from Windows, from Excel, from someone else’s editor, from an old system, from an archive. And it may contain “ugly bits” that you can’t see with your eyes but that break your logic.

“Dirty text” in the context of today’s lecture is text that is generally readable but contains hidden technical side effects. The two most common are: a BOM at the beginning (it interferes with the first word/header) and different line breaks ("\n" vs "\r\n"), which break “line-by-line” parsing if you expected one specific format.

To avoid turning parsing into fortune-telling, we’ll stick to discipline: first we make the input predictable, and only then we parse the data.

2. Cleaning plan

If, in the previous day about files, you already caught yourself thinking “why am I reading text and then still thinking about bytes?”, today that internal conflict ends with a peace treaty. We’ll honestly acknowledge: a file starts with bytes, and some problems (for example, BOM) are simpler and more reliable to solve at the byte level.

It’s useful to keep the scheme in mind as a mini pipeline:

flowchart LR
    A["readBytes()"] --> B[remove UTF-8 BOM]
    B --> C["String(bytes, UTF_8)"]
    C --> D[normalizeNewlines]
    D --> E[parse / split / comparisons]

Notice we clean twice, but at different levels: it’s more convenient to cut out BOM as bytes, and line breaks as text (because those are sequences of characters inside a string).

3. Working with bytes and the BOM

Read the file as bytes

Before you “fix” a file, it’s useful to be able to quickly check: are there any suspicious bytes in there at all? Especially in the first 3–4 bytes, where a UTF-8 BOM may be hiding. For that we need readBytes(). It returns a ByteArray—a fixed-size byte array, which we already know how to work with by index.

Mini example: “peek at the first bytes”:

import java.io.File

fun main() {
    val bytes = File("data/input.txt").readBytes()
    val preview = bytes.take(8).joinToString(separator = " ") { it.toUByte().toString() }
    println("firstBytes=$preview") // for example: firstBytes=239 187 191 105 100 44 110 97
}

Comment: I use toUByte() so the output is “human-friendly” (0..255), not negative bytes. This is purely for diagnostics.

Remove the UTF-8 BOM

Now we move on to the main “invisible culprit.” A UTF-8 BOM is three bytes at the very beginning of the file: EF BB BF (in hex). In Kotlin, a byte is signed (-128..127), so we compare using 0xEF.toByte() and so on, otherwise it gets confusing.

A careful BOM-removal function should do two things: check whether the BOM is present and only then cut off the first three bytes. If there is no BOM, it should return the original bytes unchanged. We don’t “cut just in case,” because that can eat a real first character.

fun removeUtf8Bom(bytes: ByteArray): ByteArray {
    val hasBom =
        bytes.size >= 3 &&
        bytes[0] == 0xEF.toByte() &&
        bytes[1] == 0xBB.toByte() &&
        bytes[2] == 0xBF.toByte()

    return if (hasBom) bytes.copyOfRange(3, bytes.size) else bytes
}

There are two practical nuances here.

First, copyOfRange(3, bytes.size) creates a new array and copies the data. That’s fine: we “clean the input” and get a new version. This approach is easier to understand than trying to “shift” the array in place. Also, arrays are fixed-size, so you can’t “cut off the start without copying” unless you use more elaborate constructs.

Second, we check bytes.size >= 3, otherwise accessing bytes[2] will throw due to an index issue. This is one of those rare cases where a couple of extra symbols in a condition saves your evening.

Decode into a UTF-8 string

After removing the BOM (if it was there), you can safely turn bytes into a string. And here it’s very important not to fall back to the “default encoding.” We already agreed that in a normal application, encoding is part of the contract, so you specify it explicitly.

Mini decoding function:

fun decodeUtf8(bytes: ByteArray): String {
    return String(bytes, Charsets.UTF_8)
}

And a small “before/after” check (with training data you can sometimes literally see the difference):

import java.io.File

fun main() {
    val rawBytes = File("data/input.txt").readBytes()
    val cleanBytes = removeUtf8Bom(rawBytes)

    println("rawSize=${rawBytes.size}")      // rawSize=...
    println("cleanSize=${cleanBytes.size}")  // cleanSize=... (may be smaller by 3)
}

4. Normalizing line breaks

Even if you’ve defeated the BOM, line breaks can still ruin your day. On Unix-like systems you most often see "\n" (LF). On Windows it’s usually "\r\n" (CRLF). Sometimes, from “ancient times,” you even encounter a lone "\r". Visually, in an editor you see a “new line,” but in the data these are different character sequences.

Why does this matter? Because you often do something like text.split("\n") or look for the first line using indexOf('\n'). If there’s a "\r\n" inside, your lines will have a trailing "\r", and then comparisons suddenly break: "id" is not equal to "id\r".

Line-break normalization is converting all variants into one. Most commonly, people choose "\n", because it’s shorter and more familiar in code.

fun normalizeNewlines(text: String): String {
    val step1 = text.replace("\r\n", "\n")
    return step1.replace("\r", "\n")
}

The order of replacements matters here: first "\r\n", then a lone "\r". Otherwise you can accidentally turn "\r\n" into "\n\n" (first you replace "\r" with "\n", and then the second part also yields "\n").

Mini demonstration: “why this breaks comparisons”:

fun main() {
    val windowsLine = "id,name\r\n"
    val parts = windowsLine.split("\n")

    println(parts[0] == "id,name") // false (it is actually "id,name\r")
}

And after normalization everything becomes boringly correct (and “boringly correct” is a compliment for input data):

fun main() {
    val windowsLine = "id,name\r\n"
    val normalized = normalizeNewlines(windowsLine)

    println(normalized.split("\n")[0] == "id,name") // true
}

5. The readCleanUtf8 function

Now we’ll assemble the pipeline into one convenient function so the rest of the code (parsing, commands, reports) doesn’t have to think about BOM and line breaks at all. And this is an important style choice: bring “dirty input” back to normal at the boundary, and work with clean data inside the program.

import java.io.File

fun readCleanUtf8(file: File): String {
    val rawBytes = file.readBytes()
    val bytesNoBom = removeUtf8Bom(rawBytes)

    val rawText = String(bytesNoBom, Charsets.UTF_8)
    return normalizeNewlines(rawText)
}

Pay attention to the variable names. rawBytes, bytesNoBom, rawText—this is not “wordy fluff,” it’s a way to avoid getting lost across layers. When you open the code a week later, you won’t have to guess which step you’re at and whether “we’ve already cleaned the input.”

6. Example: reading a task list

So that this lecture isn’t pure theory, let’s continue a simple console story: “we have a file with a list of tasks, one task per line.” This is very similar to how many CLI utilities store data in plain text.

Let’s imagine the file format data/tasks.txt:

• the first line is the header TASKS
• then come task lines
• empty lines are ignored

The problem we want to prevent: if the file was created by something that added a BOM, the header becomes "\uFEFFTASKS" (or a byte-equivalent), and the header check breaks.

Let’s make a function that takes text and returns a list of tasks. (Yes, it’s a List<String>— we’ve already covered collections, and they’re very much the right tool here.)

fun firstLine(text: String): String {
    val idx = text.indexOf('\n')
    return if (idx == -1) text else text.substring(0, idx)
}

Now a simple parse (note: we’re not using lists in the explanation, but in code it would be artificially hard without collections):

fun parseTasks(cleanText: String): List<String> {
    val lines = cleanText.split('\n')
    if (lines.isEmpty() || lines[0] != "TASKS") return emptyList()

    return lines.drop(1).map { it.trim() }.filter { it.isNotEmpty() }
}

And we tie everything together in main:

import java.io.File

fun main() {
    val file = File("data/tasks.txt")
    val text = readCleanUtf8(file)
    val tasks = parseTasks(text)

    println("tasksCount=${tasks.size}") // for example: tasksCount=3
}

The “magic” here is that parseTasks receives already clean text. It does not know and should not know about BOM, bytes, and "\r\n". That’s good separation of responsibility: the cleaning function is responsible for the environment, the parsing function is responsible for meaning.

7. Why it’s better to clean BOM at the byte level

Sometimes there’s a temptation: “Let’s just do after reading the text: if (text.startsWith("\uFEFF")) text.substring(1).” And that sometimes really works. But the byte approach is usually more stable for two reasons.

The first reason is that BOM is originally a byte marker. If you first decoded the bytes “somehow” (or even correctly) and then try to guess what the BOM turned into in the string, you’ve already taken a step back toward “symptoms” rather than “cause.”

The second reason is more practical: bytes allow you to say precisely “yes, this is the exact EF BB BF,” and not touch the data if it’s not a BOM. With strings, unexpected “look-alike characters” can start to appear, and debugging turns into a show of “why is my first character invisible.”

At the same time, knowing about '\uFEFF' is useful as a diagnostic trace: if you printed a string somewhere and see weird behavior in the first character, it’s a good hint. But it’s more convenient to “treat” it at the byte level.

8. Typical mistakes

Mistake #1: doing split("\n") first, and normalizing line breaks later.
This order almost guarantees you’ll get lines like "TASKS\r" or "id,name\r", after which phantom comparison bugs start. You need to normalize line breaks before any “line-by-line” splitting; otherwise you cement the problem into the data structure.

Mistake #2: “let’s cut off the first character, maybe it’s a BOM.”
This is the classic attempt to fix an unknown problem by “deleting something.” If there was no BOM, you will corrupt the data: the first letter of the header will disappear, and you’ll create a bug where there wasn’t one. A BOM is removed only after checking the signature (on bytes) or a specific "\uFEFF" at the start of the string.

Mistake #3: mixing data levels in a single variable text.
When the same variable in turn means “raw bytes,” “the string as read,” “the string after BOM removal,” “the string after normalization,” you lose control very quickly. The code starts to look short but becomes fragile: you don’t understand what state the data is in right now. Different names (rawBytes, bytesNoBom, rawText, cleanText) make the code a couple of lines longer but save hours.

Mistake #4: forgetting about bytes.size >= 3 when checking for a BOM.
Yes, files are sometimes empty. Or they contain 1–2 bytes (for example, someone created a stub). If you access bytes[2] without checking, you’ll get an index crash and will go looking for “encoding problems,” when the real problem is array bounds. And arrays, as we already know, love their bounds.

Mistake #5: assuming that if it “looks fine in the editor,” then there’s no need to encode and clean.
An editor is also a program that makes choices: which encoding to apply and how to display line breaks. Your program must be robust without the editor’s hints. So cleaning input is not paranoia—it’s a normal engineering habit.