BOM — how to recognize it and why it breaks the file

1. BOM: what it is and why it exists

If you’ve ever seen the situation “the string looks the same, but == says ‘no’”, then congratulations: you’ve met a class of problems caused by invisible characters. BOM (Byte Order Mark) is one of the most popular “ninjas” in this category. It can appear at the beginning of a text file and give no hint during normal viewing, and then suddenly break your parsing, headers, and comparisons.

BOM is a service marker, a set of special bytes at the very beginning of a file that was historically used to hint “which encoding” and/or “which byte order” (this especially applies to UTF‑16). Important: BOM is not user data; it is not a “part of the text” that a person intended to save. But if you naively read a file as a string and then compare “the first word”, BOM turns into sand in the gears.

Imagine you’re expecting a passport ("HEADER"), but you’re handed a passport with an invisible sticker over the first letter. Visually it’s the same document, but formally it’s no longer the same.

How BOM breaks the beginning of a file

In practice, BOM most often shows up like this: the first line or the first token “doesn’t match”, even though in logs or in an editor everything looks identical. This is especially painful for formats where “the first line is a header” or “the file starts with a magic signature”.

Let’s imagine our training CLI project: a task list saved to a text file. We agreed that the first line of the file must be the header:

TODO_V1

And then the task lines follow (the format isn’t important right now — it’s just text).

fun main() {
    val expectedHeader = "TODO_V1"
    val actualHeader = "\uFEFFTODO_V1"

    println(expectedHeader == actualHeader) // false
}

From a human perspective, it’s “the same thing”. From the program’s perspective, it’s not, because the leading \uFEFF makes the string different.

And then you write an honest check:

fun isValidHeader(header: String): Boolean {
    return header == "TODO_V1"
}

…yet it starts returning false “for no reason”. There is a reason. It’s just invisible — like production bugs.

How to detect BOM by bytes

When we’re talking about BOM, it’s very useful to remember the boundary: on disk there are bytes, and the string comes later. If we want to reliably recognize BOM, it makes sense to look at the first bytes of the file first. For UTF‑8, the classic BOM prefix is three bytes:

EF BB BF

To check this in Kotlin, we need a ByteArray. It’s like IntArray, but for bytes: an array of primitives, without “wrappers”. Kotlin explicitly separates such arrays into distinct types (including ByteArray).

Mini-check for a UTF‑8 BOM by the first three bytes:

fun hasUtf8Bom(bytes: ByteArray): Boolean {
    return bytes.size >= 3 &&
        bytes[0] == 0xEF.toByte() &&
        bytes[1] == 0xBB.toByte() &&
        bytes[2] == 0xBF.toByte()
}

Notice two practical points.

First, we always check bytes.size >= 3. There may be no BOM; the file may be empty; the file may contain just two bytes — and if you forget this check, you’ll get an exception of the “index out of bounds” kind, and the culprit won’t be BOM at all, but your hurry.

Second, we compare with toByte() because literals like 0xEF are Int, while the array stores Byte. Kotlin forces you to be honest here: if you want to compare, convert the types explicitly.

How BOM looks in a string: '\uFEFF'

Sometimes you don’t want to dive into bytes. For example, you’ve already read the file via readText(Charsets.UTF_8) and obtained a string, and then somewhere later the header parsing unexpectedly falls apart. In such cases, BOM can show up as the first character of the string: '\uFEFF'.

It’s important to understand: this is not “some kind of space”. And it’s not trim()’s job to remove BOM. trim() works with whitespace and line breaks, but BOM is a separate entity, and the behavior here can easily be different from what you were hoping for.

Check “is there a BOM character at the start of the string”:

fun startsWithBomChar(text: String): Boolean {
    return text.isNotEmpty() && text[0] == '\uFEFF'
}

And careful removal:

fun dropLeadingBomChar(text: String): String {
    return if (text.isNotEmpty() && text[0] == '\uFEFF') {
        text.substring(1)
    } else {
        text
    }
}

The key word here is careful. We don’t “always cut off the first character”; we do it only if we have confidently recognized the BOM character.

What kinds of BOM exist: the minimum for practice

Right now, what matters is not memorizing Unicode “by heart”, but building practical intuition: BOM is a prefix that can have different lengths. We’ll focus on the most common case (UTF‑8), but it’s useful to know that UTF‑16 also likes to start a file with service bytes.

EF BB BF

Why am I not listing all UTF‑16 variants in detail? Because the goal of the lecture is to learn to recognize the fact of BOM and its effect, not to run an “guess the endianness” exam. It’s enough for us to know: “for UTF‑8 it’s 3 bytes EF BB BF, and other encodings can also have service bytes.” This alone saves a lot of debugging time.

How to remove BOM correctly: check first

Now comes the moment where beginners often make a “sharp and elegant” move and then spend a week chasing ghosts: “let’s always drop the first 3 bytes!” — and the “BOM problem will disappear”.

Yes, it will disappear. Along with the first three bytes of real data in those files where there was no BOM. That is, you’ll fix one scenario and break ten others, just quieter ones.

The correct idea: check first, then remove.

Removing a UTF‑8 BOM at the byte level:

fun dropUtf8Bom(bytes: ByteArray): ByteArray {
    return if (hasUtf8Bom(bytes)) {
        bytes.copyOfRange(3, bytes.size)
    } else {
        bytes
    }
}

ByteArray is that very “primitive byte array” that makes these operations convenient.

Embedding BOM checking into a task project

Now let’s assemble everything into a mini story that’s as close to real life as possible. Suppose the file data/todo.txt must start with the header TODO_V1. We read the first line and compare it.

For simplicity, let’s make a function that takes the first line from the text:

fun firstLine(text: String): String {
    val idx = text.indexOf('\n')
    return if (idx == -1) text else text.substring(0, idx)
}

Now we read the file in UTF‑8 (as we agreed in the previous lecture — we set the encoding explicitly), and validate the header:

import java.io.File

fun main() {
    val text = File("data/todo.txt").readText(Charsets.UTF_8)
    val header = firstLine(text)

    println("header='$header'") // may have BOM at the start
    println(header == "TODO_V1") // false if header = "\uFEFFTODO_V1"
}

The problem here is that readText(Charsets.UTF_8) honestly read the file as UTF‑8, but if an editor or utility added a BOM, it turns into a character in the string, and you end up with an “invisible prefix”.

The “string-level” fix looks like this: clean the header of the BOM character before comparing.

import java.io.File

fun main() {
    val text = File("data/todo.txt").readText(Charsets.UTF_8)
    val header = dropLeadingBomChar(firstLine(text))

    println(header == "TODO_V1") // true
}

This is much better than “always cutting it off”, because we remove it conditionally.

Diagnostics: how to prove it’s BOM

When you suspect BOM, you really want to “see evidence”. And here we have two good techniques: look at the first bytes, or print the code of the first character of the string. Both are useful, because sometimes you read the file in a different way and the symptom changes.

First, let’s print the first 6 bytes in hex form (a minimal variant, without complex formatters):

fun firstBytesHex(bytes: ByteArray, n: Int): String {
    val count = minOf(n, bytes.size)
    return bytes.take(count).joinToString(" ") { b ->
        "%02X".format(b.toInt() and 0xFF)
    }
}

And now apply it:

import java.io.File

fun main() {
    val bytes = File("data/todo.txt").readBytes()
    println(firstBytesHex(bytes, 6)) // for example: EF BB BF 54 4F 44
}

If you see EF BB BF, it’s almost certainly a UTF‑8 BOM, and then come the bytes 54 4F 44 (that’s ASCII/UTF‑8 for T O D).

If you’re already reading a string and want to check the BOM character:

fun main() {
    val s = "\uFEFFTODO_V1"
    println(s[0].code) // 65279
}

The number 65279 is the code point U+FEFF. That’s the very “invisible passenger”.

BOM breaks not only your code

Sometimes it feels like: “Well, I’m parsing badly.” But BOM can break tools too. For example, the Kotlin changelog mentioned issues with UTF‑8 BOM in Kotlin scripting (so it’s not only you suffering; grown-ups with compilers suffer too).

This is worth remembering for two reasons. First, you stop feeling like the only person on the planet who “broke a string”. Second, you start treating BOM as a real external factor that must be accounted for when working with files, especially if files can come from different editors, OSes, and tools.

2. Typical mistakes when working with BOM

Mistake #1: “cut off the first character (or the first 3 bytes) always — and we’re done”.
This is very tempting because it looks like a quick fix. But then you start corrupting valid files without BOM. The worst part is that corruption may not be immediately noticeable: the header “got fixed”, but the first task or the first letter of the first line disappeared. You should remove BOM only after checking that it is actually present.

Mistake #2: mixing the levels “bytes vs string”.
People often do this: the file has already been read as a String, but then they try to “check the first bytes” — and build weird constructs that compare a Char with 0xEF. These are different things. If you want to reliably check a UTF‑8 BOM as EF BB BF, do it on a ByteArray. If you want to quickly clean the “invisible character” at the start of a string, check '\uFEFF'. The important thing is simply not to mix these levels in the same check.

Mistake #3: expecting trim() to remove BOM.
trim() is not a “universal mop for all invisible problems”. It is not designed to remove BOM, and even if in some environment it accidentally helped, it’s not a contract you should rely on. For BOM, it’s better to have an explicit check for '\uFEFF' or a check of the first bytes.

Mistake #4: not checking the byte array length.
A byte-level BOM check almost always starts with bytes.size >= 3 (for UTF‑8 BOM). If you forget this condition, on an empty file or a short file you’ll crash with an index error. And it will be especially annoying because the crash will happen “during diagnostics”, not in the real logic.

Mistake #5: trying to cure BOM by “picking a different encoding”.
Sometimes people try: “let’s read UTF‑16… let’s read ASCII…”. That can turn the text into mojibake, but BOM won’t disappear as a phenomenon. BOM is a separate layer of the problem: a service prefix in the file. The encoding matters, but BOM must be either handled correctly on input or removed as a service marker — not “guessed” over and over until it looks pretty.