1. Why you need bufio.Scanner at all, and what a token is
When you first see Scanner, it feels too convenient—almost suspicious: the loop looks clean, there are no “errors on every iteration,” and everything reads like a normal collection iteration. That’s the point: Scanner is a “tokenizer on top of a stream.” It takes an io.Reader, reads data in chunks on its own, and gives you tokens: lines, words, runes, or anything else if you provide your own splitting logic.
In Scanner terminology, a token is “a piece of data you consider atomic for processing.” By default, a token is a line (split by \n). You can switch to words, runes, etc. But it’s important to remember: a token must fit entirely into the internal buffer. If it doesn’t, scanning stops with an error. That’s the central topic of this lecture.
The canonical pattern: Scan() in a loop and Err() after the loop
If Go handed out a medal “for the most recognizable loop,” Scanner would definitely make the finals. And this pattern isn’t just “how it’s done.” It reflects the API design: Scan() returns a bool, and you fetch the error via a separate Err() call after the loop finishes.
This is deliberate design: the main control flow (“while there are tokens—process them”) doesn’t get cluttered with error checks at every step.
An example that reads lines from any io.Reader:
package main
import (
"bufio"
"fmt"
"strings"
)
func main() {
sc := bufio.NewScanner(strings.NewReader("a\nb\nc\n"))
for sc.Scan() {
fmt.Println("token:", sc.Text()) // token: a ...
}
fmt.Println("err:", sc.Err()) // err: <nil>
}
The documentation puts it like this: Scan() returns false when tokens are exhausted due to EOF or an error; after that, Err() returns the error (except io.EOF, which is turned into nil).
Get used to the discipline: the loop ended—check Err(). Even if it feels like “we’re reading from a string, what errors could there be?”
2. Token size limit and the token too long error
Why the limit exists at all
Scanner can read lines, words, etc., but it’s not meant to keep an infinitely large token in memory. That’s why bufio has the constant MaxScanTokenSize = 64 * 1024 (i.e., 64 KiB), which is used as the default maximum token buffer size.
Practically, this means: if you’re reading lines (the default mode) and you encounter a line longer than ~64 KiB, Scanner will stop. And it can happen “suddenly” on the 10,000th line if that’s where someone put a huge JSON on a single line or just “accidentally wrote a log with no line breaks.”
The reason is pragmatic. If Scanner automatically expanded its buffer “until it wins,” a single malicious (or just sloppy) input could force the program to allocate a massive amount of memory. In CLI tools, that easily becomes “one file takes down the program.” So the limit is essentially a seat belt.
What the “token too long” error looks like
Scanner errors are pretty straightforward: if the token doesn’t fit, you get bufio.ErrTooLong, which is defined in the package as "bufio.Scanner: token too long".
Let’s reproduce it on purpose. In real life you wouldn’t do this (although log files sometimes try):
package main
import (
"bufio"
"fmt"
"strings"
)
func main() {
longLine := strings.Repeat("x", 70*1024) // 70 KiB
sc := bufio.NewScanner(strings.NewReader(longLine))
ok := sc.Scan()
fmt.Println("scan ok:", ok) // scan ok: false
fmt.Println("err:", sc.Err()) // err: bufio.Scanner: token too long
}
Why did ok become false? Because Scan() says: “I can’t produce a token,” and stops scanning. And why no panic? Because this is not a programmer error; it’s an input characteristic you should be able to handle.
By the way, the documentation highlights another unpleasant nuance: when scanning stops due to an error (or due to an oversized token), the internal Reader may have already advanced quite far in the input. So this is not a tool where you can reliably “rewind a bit and reread.” If you need that kind of precision—use bufio.Reader.
4. Workarounds and alternatives
Increase the limit via Scanner.Buffer
The most common way to “fix” the problem is to tell Scanner: “okay, I expect long tokens, here’s a higher limit.” For this, there’s the method Buffer(buf []byte, max int). It sets the initial buffer and the maximum size up to which Scanner is allowed to grow.
There are two rules worth remembering.
First: you must call Buffer(...) before scanning starts. If you call it after the first Scan(), it will panic—the documentation states this explicitly.
Second: the maximum token size must be no greater than the larger of max and cap(buf); otherwise Scanner can’t guarantee buffering.
A practical example: allow lines up to 1 MiB.
package main
import (
"bufio"
"fmt"
"strings"
)
func main() {
longLine := strings.Repeat("x", 200*1024) // 200 KiB
sc := bufio.NewScanner(strings.NewReader(longLine))
sc.Buffer(make([]byte, 1024), 1024*1024) // start 1 KiB, max 1 MiB
fmt.Println(sc.Scan()) // true
fmt.Println(len(sc.Text())) // 204800
}
This setting makes your program more resilient to “accidentally long lines.” But don’t turn it into “set 1 GB just in case.” If the input can truly be huge, it’s better to change the reading approach rather than trying to swallow “one line the size of a movie.”
Change the splitting strategy
Sometimes the problem isn’t that the input is huge, but that you chose the wrong processing unit. If you read “by lines” and a line can be enormous, maybe you don’t need a line as an atomic object at all. Maybe you care about words, or individual bytes, or runes.
bufio includes ready-made split functions: ScanLines, ScanWords, ScanRunes, ScanBytes. The documentation, for example, clarifies that ScanLines returns lines without the \n, and the last line may be returned even without a trailing newline.
In terms of the “token limit,” it works like this: if you switch to ScanWords and you don’t have words longer than 64 KiB, you suddenly stop hitting the limit. But if you have “one word of 500 KiB” (for example, base64 with no spaces), the problem remains.
Mini example with words:
package main
import (
"bufio"
"fmt"
"strings"
)
func main() {
sc := bufio.NewScanner(strings.NewReader("go is fun"))
sc.Split(bufio.ScanWords)
for sc.Scan() {
fmt.Println(sc.Text()) // go / is / fun
}
fmt.Println("err:", sc.Err()) // err: <nil>
}
This isn’t a silver bullet, but sometimes correct tokenization solves the problem better than increasing the buffer.
Switch to bufio.Reader if you need control
There are situations where Scanner is simply not your tool. And that’s fine: Go has many “simple tools” that honestly say, “I’m convenient, but not universal.” The bufio documentation directly recommends: if you need large tokens, more control over errors, or sequential scans of the same Reader, use bufio.Reader instead of Scanner.
Why is bufio.Reader better in these cases? Because it lets you read the stream “up to a delimiter” (ReadString, ReadBytes) and you decide what to do if a line is long. Yes, the code becomes a bit more “manual,” but you get fewer surprises.
Mini example: read “lines” via ReadString('\n'). It’s important to remember that at end of file you can get a partial line and io.EOF at the same time—and that’s not an “error,” it’s a normal ending.
package main
import (
"bufio"
"fmt"
"io"
"strings"
)
func main() {
r := bufio.NewReader(strings.NewReader("a\nb\nlast"))
for {
s, err := r.ReadString('\n')
if len(s) > 0 {
fmt.Printf("line: %q\n", s)
}
if err == io.EOF {
break
}
if err != nil {
fmt.Println("read error:", err)
return
}
}
}
This approach is especially good when “lines” can be large, but you’re willing to process them as you read, without relying on Scanner’s internal limits.
5. Example from an app: task import and long descriptions
Let’s imagine our educational CLI task app (let’s call it tasker). Up to this point we’ve already learned how to open files and read/write them, and now we want to import tasks from a simple text format: one task—one line. Sounds like a perfect use case for Scanner.
But a user might have a task like “paste a 120,000-character spec here”—and that one line suddenly breaks the import. So we’ll build in protection right away: increase the token limit to a reasonable value.
Let’s make a function that accepts an io.Reader (not a filename). This is a good habit: we separate “where the data comes from” from “how we parse it.” This style later lets you read from a file, from stdin, or from a string.
package main
import (
"bufio"
"fmt"
"io"
)
type Task struct {
ID int
Title string
}
func ReadTasks(r io.Reader) ([]Task, error) {
sc := bufio.NewScanner(r)
sc.Buffer(make([]byte, 1024), 1024*1024) // up to 1 MiB per line
var tasks []Task
id := 1
for sc.Scan() {
tasks = append(tasks, Task{ID: id, Title: sc.Text()})
id++
}
return tasks, sc.Err()
}
Notice two things.
First, we return sc.Err() as the function error. That’s exactly the contract you should follow: no printing inside, no fmt.Println("oops"). Just return the error upward.
Second, we didn’t forget that Err() is checked after the loop. This isn’t the author’s “special opinion”—it’s the basic mechanics of Scanner: the error is stored internally and extracted separately.
Now an example usage (no files—just to see the behavior):
package main
import (
"fmt"
"strings"
)
func main() {
input := "buy milk\nwrite code\n"
tasks, err := ReadTasks(strings.NewReader(input))
fmt.Println("err:", err) // err: <nil>
fmt.Println("count:", len(tasks)) // count: 2
}
If in the future you decide that “lines can be very long, and memory is precious,” you already have plan B: rewrite ReadTasks to use bufio.Reader.ReadString('\n'). That will be a local replacement inside one function, not a rewrite of the whole program.
6. Common mistakes when using bufio.Scanner with large tokens
Mistake No. 1: not checking scanner.Err() after the loop.
The most insidious part is that Scanner “pretends everything is fine” until you ask “are you sure?” Scan() simply returns false, and if you don’t call Err(), you may mistake a “read error” for “the file ended.” The correct pattern is for Scan() and then a mandatory Err().
Mistake No. 2: increasing the limit but calling Buffer() too late.
You can’t call Buffer() “when things already went bad.” The method must be called before scanning starts; otherwise there will be a panic. This isn’t nitpicking: the scanner’s internal state already uses the buffer, and changing the rules mid-stream breaks invariants.
Mistake No. 3: treating any input by setting max = a billion.
Yes, this can “defeat” the token too long error. But at the same time, you allow the input to force the program to allocate a lot of memory. Usually it’s better to set a reasonable limit (for example, 1–10 MiB depending on the task) and, when exceeded, honestly return bufio.ErrTooLong. That error exists precisely so you can distinguish it.
Mistake No. 4: trying to use Scanner where control over reading is required.
If you need to read gigantic lines, do “multiple passes” over a single Reader, or very precisely control where reading stopped, Scanner may be an inconvenient choice. The documentation explicitly says that for large tokens and more control you should prefer bufio.Reader.
GO TO FULL VERSION