CSV nuances — quotes, delimiters, header, BOM

1. CSV as a format: records, fields, and line breaks

When you first see CSV, you really want to say: “Come on, it’s just strings.Split(line, ",")—what is there to study?” And right at that moment CSV quietly pulls out quotes and newlines and says: “Surprise.” CSV is a sequence of records, and each record consists of fields. On paper everything is simple, but real-world CSV is an agreement that allows tricky cases.

The main thing to lock in right away: a line in a file does not have to match a CSV record. Yes, records are usually separated by line breaks. But if a field is quoted, then a line break inside the quotes is considered part of the value. So “read the file line by line” and “read CSV records” are two different actions.

Imagine our learning project: a small task tracker (todo list). We want to store tasks in tasks.csv roughly like this:

As long as note has no commas or line breaks, everything looks innocent. But once the user writes a note like “buy apples, bananas, and milk”, the comma suddenly becomes part of the data rather than a separator.

Why strings.Split breaks CSV

Below is an example of an “innocent” CSV where a comma inside a field makes Split useless:

package main

import (
	"fmt"
	"strings"
)

func main() {
	line := `2,"Buy apples, bananas",false`
	parts := strings.Split(line, ",")
	fmt.Println(parts) // [2 "Buy apples  bananas" false]
}

The output comment here is intentionally “wonky”: you will see that Split cuts at every comma and does not understand quotes (which means it breaks the record structure). And this is not a “rare corner of the format”—this is everyday reality.

Why a file line is not always equal to a record

To reinforce the idea, here is a diagram of why a “file line” may not match a “CSV record”:

flowchart TD
    A["File (bytes)"] --> B["Line breaks (LF)"]
    B --> C["File lines"]
    A --> D["CSV parser"]
    D --> E["Records (records)"]
    E --> F["Fields (fields)"]
    C -. "not always 1:1" .-> E

2. Delimiter and spaces around it

Almost everyone knows CSV as “Comma-Separated Values”. But the industry (and accounting) can surprise you: very often the delimiter is ; (semicolon), especially in locales where a comma is used as the decimal separator in numbers. And then “CSV” is still CSV in spirit, but the “C” in the name kind of lies to your face.

The right mindset is important here: the delimiter is part of the input file format, not something we guess “based on vibes”. If you expect ; but the file arrives with ,, you should either honestly report an error, or explicitly support both variants according to product rules. But “let’s just Split by comma” is about as reliable as fixing a server with a palm slap (sometimes it works, but you do not want it to become a company standard).

A mini example that shows the problem plainly: two lines look almost the same, but the meaning changes because of the delimiter.

package main

import "fmt"

func main() {
	fmt.Println("id;title;done") // id;title;done
	fmt.Println("id,title,done") // id,title,done
}

For a human, it “looks similar”. For code, these are two different dialects. And a good import has to take that into account.

One more nuance: spaces. CSV files often contain spaces after the delimiter, especially if someone edited them by hand:

id, title, done
1, Buy milk, false

It is convenient for a human. For strict parsing this can be a separate policy: treat the space as part of the data or ignore it. The only important thing here is to acknowledge that this kind of input exists, and “manual Split” does not give you clean, consistent behavior.

3. Quotes and escaping

When a field must be quoted

Quotes in CSV follow a simple rule: quotes appear when a field cannot be safely written “as is”. The most common reasons are: the field contains a delimiter, a double quote, or a line break. Then the value is wrapped in double quotes "...", and the parser understands: “this is one chunk of data—do not cut it by commas.”

A useful everyday analogy: quotes in CSV are like a box for fragile goods. As long as you have “normal items”, you can carry them without a box. But once you have “glass” inside (comma/line break/quote), the box becomes mandatory—otherwise everything will shatter on the way.

Back to our todo file example. Let note contain a comma. Then a sensible CSV line looks like this:

2,"Buy apples, bananas",false,"for smoothie"

And if the note contains a line break (for example, the user writes a list), CSV might look like this:

3,"Plan trip",false,"step1: tickets
step2: hotel"

And this is where many beginners “break”: they see two lines in the file and think those are two records. But this is one record; the field simply contains a line break.

Let’s show this effect with a small Go example. It does not parse CSV; it just demonstrates that a line break inside quotes is perfectly normal text.

package main

import "fmt"

func main() {
	data := "id,note\n1,\"line1\nline2\"\n"
	fmt.Print(data)
	// id,note
	// 1,"line1
	// line2"
}

What matters here is not that the output looks “pretty”, but that a line break can genuinely live inside a single value. If you read the file “by lines” and try to run Split on each line, you will saw one field into two—and then wonder why your columns suddenly “shifted”.

How a quote inside a value is encoded

When a double quote " appears in a field, CSV does not use a backslash like many languages do (\"). CSV uses a different rule: a double quote inside a field is encoded as two double quotes in a row. That is:

• value: He said "hi"
• CSV field representation: "He said ""hi"""

Yes, it looks like the quotes reproduce by budding. But this is standard CSV logic: “a quote opens/closes the field, and a doubled quote inside is just a quote character.”

A small example with a string literal in Go so you get used to it visually. (And yes, this is the case where you start respecting raw strings in backticks.)

package main

import "fmt"

func main() {
	field := `"He said ""hi"""`
	fmt.Println(field) // "He said ""hi"""
}

Note: this is a string that contains quote characters. We are not parsing yet—we are just learning to recognize such fragments by sight, because you will encounter them in real files.

4. Empty fields and structural stability

In CSV, an empty field is not an error and not a “space”. It is a normal situation that must be preserved as a fact. For example, the line:

a,b,

contains three fields, not two. The third field is empty. This matters because an empty field can mean “no note”, “unknown”, “not provided by the user”, “value will be computed later”, and so on. In any case, it is not something you should lose.

Why is this especially sensitive for our todo project? Because if you export tasks and a task has no note, you still want to preserve the number of columns and the position of the note column, otherwise, on import, “indexes will drift”. CSV is about order and regularity: if you have four columns in the header, you also expect four fields in the rows (even if some are empty).

5. Header and BOM

The header as a practical contract

A CSV header is the first record where, instead of data, you write column names: id,title,done,note. From the CSV format point of view, it is just another record. But from an engineering point of view, it is an agreement that saves you hours of life and liters of coffee.

If you tie yourself to indexes (“the second column is title”), any format change turns into a minefield. Add a created_at column in the middle—and that’s it: your import starts reading done as note, and note as “well, I have no idea what this is”. If you use a header, you take a grown-up step: you map “column name → index” and then work by name.

Even if you are not writing an import right now, it is useful to understand what this looks like logically:

flowchart TD
    A["Read header: [id title done note]"] --> B["Build map: name -> index"]
    B --> C["Then read data rows"]
    C --> D["Take required fields by name (via index)"]

So the header is like a table of contents in a book: you can, of course, read everything in order and search with your eyes, but the table of contents makes life easier and reduces the chance you suddenly end up in the chapter “About dragons” instead of “About payments”.

BOM: an invisible symbol in the first column

A BOM is a small technical thing that can appear at the beginning of a UTF-8 file. Most of the time you do not need a BOM, but some programs (especially from the office world) like to add it “just in case”. And if a BOM is present, it sticks to the beginning of the first field. In CSV it almost always breaks the header: instead of "id" you get "\ufeffid".

The problem is that you will barely see it by eye. In logs, a regular fmt.Println often does not help either. So it is useful to remember the %q format, which prints a string with escaping.

In Go, the BOM character is convenient to represent as the string "\ufeff". It is just a normal string literal, only with a Unicode code point.

Demonstration:

package main

import (
	"fmt"
	"strings"
)

func main() {
	first := "\ufeffid"        // BOM + "id"
	fmt.Println(first == "id") // false

	fmt.Printf("%q\n", first) // "\ufeffid"

	fmt.Println(strings.TrimPrefix(first, "\ufeff") == "id") // true
}

There are two key ideas here. First: string comparison breaks because it is actually a different sequence of characters. Second: normalizing the header (at least the first field) is not a “hack”, but a very practical step if your CSV files come from the outside world.

6. CSV mini cheat sheet

Before the finale, let’s assemble a mini cheat sheet: a table is more convenient when, a week later, you open someone else’s CSV and quietly sigh.

7. Common mistakes

Mistake #1: parse CSV via strings.Split and consider the task done.
Usually this works exactly until the first “inconvenient” value: a comma in text, quotes in a note, or a line break. After that, the import either falls apart or (worse) starts silently corrupting data. CSV was designed as a format with rules, so it needs a parser that knows those rules.

Mistake #2: think a line break always means a new record.
Once a field is quoted, a line break inside the quotes is part of the value. So “read the file line by line” is not the same as “read CSV records”. This is especially common when CSV is exported from systems that store multiline text in comments or descriptions.

Mistake #3: ignore empty fields and “collapse” the tail.
The line a,b, contains three fields. If you turn it into “two fields because the third is empty”, you lose information and break the alignment with the header columns. Emptiness is also a value—just an “empty” one.

Mistake #4: rely on column indexes and ignore the header.
As long as the format has not changed, it feels like “column 0 is id” is reliable. Then a new export arrives with an additional column, and everything falls apart. A header provides a stable “name → index” contract and makes the import much more resilient.

Mistake #5: do not account for BOM, then spend an hour looking for why id is “not found”.
A BOM is almost invisible visually, but in strings it is a real character. As a result, "\ufeffid" != "id", and your column mapping breaks at the very first step. Normalizing the first header via strings.TrimPrefix(header[0], "\ufeff") often saves you from very strange bugs.