Buffering and chunk copying

1. Copying one byte at a time is slow

If you have ever tried to carry water with a teaspoon when there is a bucket nearby, you already get the idea of this lecture. Reading and writing one byte at a time is technically possible, but it almost always leads to a huge number of calls into the I/O system. And I/O operations are not a “tiny plus” in memory; they are a trip “outside” (to the OS, the file system, the disk).

Naive code like “read a byte → immediately write a byte” most often loses in performance by multiple times, sometimes by an order of magnitude. The reason is not that the computer “doesn’t like bytes”, but that it doesn’t like millions of small I/O operations. We want fewer operations—and to move data in blocks, for example 8 KB at a time.

A mini sketch of “how NOT to do it” (this is not a “forbidden technique”, it’s “have mercy on your SSD”):

import java.io.FileInputStream
import java.io.FileOutputStream

fun main() {
    FileInputStream("in.bin").use { input ->
        FileOutputStream("out.bin").use { output ->
            while (true) {
                val b = input.read()
                if (b == -1) break
                output.write(b)
            }
        }
    }
}

The code is correct in meaning, but the data portions are too “tiny”.

2. Buffer vs buffering: ByteArray vs Buffered...Stream

Now for an important point that saves you from about half of the typical mistakes in this topic. When we say “buffer”, we may mean two different things that are often used together, but solve different problems.

First: our buffer is a regular ByteArray that we read bytes into and write bytes from. It’s just a fixed-size container. The familiar array logic applies here: indexes, size, and so on. ByteArray is a primitive array in Kotlin (no boxing), which is especially appropriate for “byte-for-byte” data.

Second: stream buffering is wrappers like BufferedInputStream and BufferedOutputStream that keep an additional buffer internally and reduce the number of calls to the underlying stream. This is an optimization “one level lower”: even if you read in blocks, a buffered stream can do it even more efficiently.

It’s convenient to keep the difference in a single table:

ByteArray buffer

BufferedInputStream/BufferedOutputStream

So: our ByteArray is the “bowl”, and Buffered...Stream is the “waiter who brings a whole tray at once instead of a single cutlet”.

3. The read(buffer) contract: the n value

For chunk copying to work correctly, you need to accept a simple but very important idea: a call to read(buffer) is not required to fill the buffer completely.

The method read(buffer) returns an Int, and that number means: “how many bytes were actually read and placed into buffer”. The return value:

• can be less than the buffer size, even if the file is not finished yet;
• will be -1 when EOF (end of stream) is reached.

And this is where the key copying invariant is born: you must write exactly n bytes to the output stream, not the entire buffer.

Let’s see the idea on a very small example (no files): we’ll read from a ByteArrayInputStream in chunks of 4 bytes. Yes, it’s a “toy” source, but it demonstrates the contract perfectly.

import java.io.ByteArrayInputStream

fun main() {
    val data = byteArrayOf(10, 20, 30, 40, 50)
    val input = ByteArrayInputStream(data)

    val buffer = ByteArray(4)
    val n = input.read(buffer)

    println("n=$n")          // n=4
    println(buffer[0])       // 10
}

If you do a second read, it will return 1, because there is only one byte left. And that’s normal. That’s exactly why you cannot ignore n.

4. The canonical chunk-copy loop

Here is the “golden template” that you should not only understand, but get used to writing almost automatically. It’s simple:

1. Create a fixed-size ByteArray (for example, 8 * 1024).
2. In a loop, read n = input.read(buffer).
3. If n == -1, exit.
4. Otherwise, write output.write(buffer, 0, n).

Here is the minimal correct version as a function (this is more convenient than keeping copying directly in main):

import java.io.BufferedInputStream
import java.io.BufferedOutputStream
import java.io.FileInputStream
import java.io.FileOutputStream

fun copyFileChunked(srcPath: String, dstPath: String) {
    FileInputStream(srcPath).use { rawIn ->
        FileOutputStream(dstPath).use { rawOut ->
            BufferedInputStream(rawIn).use { input ->
                BufferedOutputStream(rawOut).use { output ->
                    val buffer = ByteArray(8 * 1024)

                    while (true) {
                        val n = input.read(buffer)
                        if (n == -1) break
                        output.write(buffer, 0, n)
                    }
                }
            }
        }
    }
}

Notice two things. First, we close resources via use {}—that’s the idiomatic Kotlin approach for anything that implements AutoCloseable: the stream will be closed even if an error occurs inside. Second, we use Buffered...Stream, but we do not replace our buffer with them: we still read into a ByteArray.

5. Why output.write(buffer) is a common mistake

This section often surprises beginners at the level of: “But I filled the buffer, why not write all of it?”. Let’s break down why you can’t do that.

If you write output.write(buffer), you write the entire array—that is, buffer.size bytes. But read(buffer) may have read only n bytes. The remaining elements in the buffer either stayed from the previous loop iteration, or contain “old garbage” (meaning previous values in the array). Result: at the end of the file you will append extra bytes, and the copy will be corrupted.

A bad example (the mistake is intentional):

import java.io.FileInputStream
import java.io.FileOutputStream

fun main() {
    FileInputStream("in.bin").use { input ->
        FileOutputStream("out.bin").use { output ->
            val buffer = ByteArray(1024)
            val n = input.read(buffer)
            if (n != -1) output.write(buffer) // ERROR: writes all 1024 bytes
        }
    }
}

The correct version: write(buffer, 0, n).

import java.io.FileInputStream
import java.io.FileOutputStream

fun main() {
    FileInputStream("in.bin").use { input ->
        FileOutputStream("out.bin").use { output ->
            val buffer = ByteArray(1024)
            val n = input.read(buffer)
            if (n != -1) output.write(buffer, 0, n) // write only what was read
        }
    }
}

Why does the mistake sometimes “not show up”? Because if the file size is exactly a multiple of the buffer size, then the last block will also be full and the bug will hide. It’s like a “crashes only on Fridays” bug—an especially beloved genre among developers.

6. Embedding copying into a mini utility

A simple main without extra magic

So the examples are not just a set of separate snippets, let’s imagine we are growing a simple console utility, let’s call it FileTool. Its goal is to do one thing: copy a file from A to B.

For now: no complex commands, no argument parsing, and no fancy UX (today’s topic is not CLI architecture). We just take two paths and start copying.

import java.io.File

fun main() {
    val src = File("in.bin")
    val dst = File("out.bin")

    copyFileChunked(src.path, dst.path)
    println("Done. size=${dst.length()} bytes") // Done. size=...
}

In this version, we rely on the fact that copyFileChunked already closes streams correctly via use. And yes, File.length() is a simple first check: the file was copied and has some size. But “some size” is not a guarantee that the data is correct; it’s just a quick signal that “we definitely wrote something”.

Quick check: size and contentEquals

It’s very tempting to say “done” after copying and go make tea. But sometimes it’s useful to at least minimally verify that the copy resembles the original.

The fastest and cheapest test is to compare sizes:

import java.io.File

fun main() {
    val src = File("in.bin")
    val dst = File("out.bin")

    println(src.length() == dst.length()) // true/false
}

But equal size does not prove the bytes match: two different files can easily have the same size. If the file is small, you can compare its contents in memory. The key is not to try this for gigabyte-sized files.

Array comparison in Kotlin is done via contentEquals, not via ==.

import java.io.File

fun main() {
    val a = File("in.bin").readBytes()
    val b = File("out.bin").readBytes()

    println(a.contentEquals(b)) // true/false
}

This check is “byte for byte”, but at the cost of loading both files into memory. So it’s good as a test on small examples and in learning scenarios, but not as a universal strategy “for everything in the world”.

7. Buffer size and the data flow diagram

Buffer size: reasonable values

People love asking about buffer size as if there were one “correct size for all time”. Spoiler: there isn’t. There are reasonable values that usually work well, and there are extremes that usually don’t help.

Why do people often choose 8 * 1024 (8 KB)? Because it’s a good compromise: a large enough chunk to avoid too many iterations, and small enough not to bloat memory or complicate cache behavior. In practice, you will often see 4 KB, 8 KB, 16 KB, 64 KB—all of these can be perfectly fine.

If you make the buffer too small (for example, 1 byte or 16 bytes), you will lose performance due to the number of read/write calls. If you make the buffer too large (for example, tens of megabytes), you’ll start wasting memory and might not gain anything: you’ll be limited by disk/file system speed, plus a giant array is not as friendly to caches and garbage collection.

A useful habit: define the buffer size as a constant so it is “in one place” and reads well.

private const val DEFAULT_BUFFER_SIZE_BYTES = 8 * 1024

fun makeBuffer(): ByteArray = ByteArray(DEFAULT_BUFFER_SIZE_BYTES)

And then use makeBuffer() in copying. The code will become a bit longer, but the meaning of “why 8192” will be more obvious.

Data flow diagram in chunk copying

So you don’t feel like “I just copied a template and I’m praying”, it’s useful to see the process diagram once.

flowchart TD
    A[Source file] --> B[FileInputStream]
    B --> C[BufferedInputStream]
    C --> D[ByteArray buffer]
    D --> E[BufferedOutputStream]
    E --> F[FileOutputStream]
    F --> G[Destination file]

Key idea: ByteArray buffer is not an “in-memory archive”, it’s just a work area for a single loop iteration. We read a chunk, write a chunk, and reuse the same array over and over.

8. Common mistakes in chunk copying

Mistake #1: ignore n and write the whole buffer.
This is the most common problem: read(buffer) read 137 bytes, but you wrote 8192. As a result, a “tail” of old buffer values gets appended to the end of the file. This mistake is especially tricky because on some files it won’t show up if the file size is a multiple of the buffer size, but on others it will break the copy for sure.

Mistake #2: assume a single read(buffer) call is enough to copy.
Streams are not required to return the entire file in one read call. Even if the file is small, the correct model is “read portions until EOF”. One read is, at best, “read the beginning”.

Mistake #3: confuse “our buffer” with BufferedInputStream.
Sometimes beginners think that if they wrapped the stream in BufferedInputStream, then they no longer need a ByteArray. But BufferedInputStream does not eliminate your task of reading in chunks and writing in chunks. It only makes it more efficient internally by reducing calls to the underlying stream.

Mistake #4: not closing streams (or closing them “somehow”).
File streams are resources, and you must close them reliably. In Kotlin, use {} exists for this: inside the block you work with the resource, and after you exit it is closed automatically, even if an exception occurs. If you don’t close streams, you can get unwritten data, descriptor leaks, and very strange “sometimes it works, sometimes it doesn’t”.

Mistake #5: compare arrays using ==.
If you write tests and want to compare two ByteArray values, use contentEquals. In Kotlin, == for arrays checks not the contents, but whether “this is the same object or not”. For content comparison, contentEquals exists.

Mistake #6: try to validate the copy by “opening it in Notepad”.
A binary file in a text editor almost always looks like gibberish, and that’s normal. Visual “opened it, looks similar” checks do not work here. For small files, use contentEquals; for large files, at least compare sizes; even better, use dedicated verification strategies (but that’s a separate engineering topic, and we won’t go there today).