1. Sometimes you want to process files in parallel
When you’re just starting to program, it seems logical: “well, it’s one file after another — read it, process it, move on.” And that does work… as long as there aren’t many files and they’re small. But in real life, the scenario is often “I have a folder of logs / photos / reports / sources” and you need to do the same thing for each file: compute size, count lines, validate the format, extract metadata, copy, rename, generate a report.
And that’s when the natural desire appears: if there are many files, why not do several at the same time? Coroutines are designed exactly for “starting many tasks” without manually spawning threads (which then makes you want to hide under the desk). Kotlin explicitly describes coroutines as a lightweight concurrency mechanism that lets you suspend execution without blocking system resources.
But there’s a catch: “in parallel” does not mean “as many as possible at once.” The file system and the disk aren’t infinite. So this lecture has two layers: how to parallelize, and how not to overdo it.
2. launch and async: similar on the outside, different in meaning
At this point, beginners usually start suspecting that programmers intentionally invent similar words to make you suffer. And… yes, sometimes it does look that way.
Both launch and async are coroutine builders, i.e., functions that start a coroutine. Kotlin documentation explicitly lists them as the basic coroutine builders. But their meaning differs:
- launch — “start it and take responsibility as a Job”: a coroutine for an action.
- async — “start it and give me the result later”: a coroutine for a result; it returns Deferred<T>.
To make it easier on your brain, here’s a small table:
| What you need | What to use | What it returns | When the error shows up |
|---|---|---|---|
| “Just do it” | |
|
usually “inside” (or via handlers) |
| “Do it and return a value” | |
|
on await() / awaitAll() |
Mini example: running an action with launch
import kotlinx.coroutines.launch
import kotlinx.coroutines.runBlocking
fun main() = runBlocking {
val job = launch {
println("I'm just doing something") // I'm just doing something
}
job.join()
}
Mini example: running a computation and waiting for the result with async
import kotlinx.coroutines.async
import kotlinx.coroutines.runBlocking
fun main() = runBlocking {
val d = async { 2 + 3 }
println(d.await()) // 5
}
In this lecture we’re specifically interested in async, because when processing files we almost always need a result: file size, stats, text, hash, report — anything.
3. Practical example: the File Reporter utility
So the examples aren’t just a set of unrelated spells, let’s imagine we’re writing a console utility called File Reporter. It takes a list of files and produces a short report: name, size, number of lines (for text files).
We’ll keep the data model simple, without “future-proof” OOP tricks. Just a data class, because we want to store the result conveniently.
import java.io.File
data class FileReport(
val file: File,
val sizeBytes: Long,
val lineCount: Int
)
Now let’s make a suspend function that processes a single file. Important: file reading goes to Dispatchers.IO, and inside we use .use {}.
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
import java.io.File
suspend fun buildReport(file: File): FileReport =
withContext(Dispatchers.IO) {
val size = file.length()
val lines = file.bufferedReader().use { it.lineSequence().count() }
FileReport(file = file, sizeBytes = size, lineCount = lines)
}
Yes, lineSequence().count() inside looks pretty “clever”, but conceptually it’s simple: read lines and count them.
4. Pattern: many files → many async → one awaitAll()
The nicest part of today’s lecture: once you see this pattern, you start noticing it in code everywhere. It’s like a “pipeline”: we create a batch of tasks and then wait for all of them to finish.
Technically it looks like this:
- we have a List<File>
- we do map { async { ... } } and get a List<Deferred<FileReport>>
- we call awaitAll() and get a List<FileReport>
awaitAll() is convenient because it “collects results in a batch”, and that makes the code much simpler than manually using for plus await().
Example: build reports for a list of files in parallel
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.async
import kotlinx.coroutines.awaitAll
import kotlinx.coroutines.runBlocking
import java.io.File
fun main() = runBlocking {
val files = listOf(File("a.txt"), File("b.txt"), File("c.txt"))
val reports = files
.map { f -> async(Dispatchers.IO) { buildReport(f) } }
.awaitAll()
println("reports=${reports.size}") // reports=3
}
Notice a small detail: we passed Dispatchers.IO directly into async(...). That means the coroutine for each file runs in the I/O context (so blocking file reads are acceptable). This fits the overall “files in IO” model we adopted earlier.
And yes: Kotlin documentation emphasizes that launch() and async() are basic coroutine builders that start concurrent tasks inside a CoroutineScope.
5. Why “a thousand async” isn’t always a party
At some point, a student does this: “I have 20,000 files. I’ll just do .map { async { ... } }.awaitAll().” And at that moment the computer starts sounding like a small airport, and you learn what “performance degradation from too much enthusiasm” means.
The problem isn’t coroutines — coroutines are indeed lightweight. The problem is that:
- each file read still performs blocking OS calls;
- the file system and disk won’t get faster just because you requested “20,000 operations at once”;
- you create load on the scheduler, I/O queues, caches, and everything between you and the disk.
So we make the following “engineering” move: limit parallelism.
Dispatchers.IO.limitedParallelism(n)
The idea is simple: we take the I/O dispatcher and say: “dear dispatcher, please don’t run more than n tasks at the same time.” The rest should wait their turn.
import kotlinx.coroutines.Dispatchers
val io4 = Dispatchers.IO.limitedParallelism(4)
Then we start all async on io4.
Example: limit concurrency to 4 files
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.async
import kotlinx.coroutines.awaitAll
import kotlinx.coroutines.runBlocking
import java.io.File
fun main() = runBlocking {
val io4 = Dispatchers.IO.limitedParallelism(4)
val files = (1..10).map { File("file-$it.txt") }
val reports = files
.map { f -> async(io4) { buildReport(f) } }
.awaitAll()
println("done=${reports.size}") // done=10
}
Note that there are 10 files and parallelism is 4. That doesn’t mean we’ll process only 4 files. It means that at most 4 tasks will be “in progress” at the same time, and the rest will wait until a slot is free.
How to choose n
There’s no “eternally correct” number, and that’s fine. In a learning project, 2–8 is usually enough. If you set 64 “because it looks nice”, you most likely won’t get a speedup, and sometimes you’ll even slow things down. In real projects, parallelism is chosen experimentally and/or configured.
6. The I/O vs CPU boundary: why it affects speed
When you read a file, you do I/O. But often after reading comes processing: parsing, searching, counting, text normalization, computing a hash. And that’s CPU work.
A naive solution: “I’ll do everything inside Dispatchers.IO, since the file is there.” It will work, but architecturally it’s blurry: you’re mixing different types of load.
It’s useful to keep a rule: inside the I/O boundary, do only what’s related to the file system and streams. Everything else is regular computation, and it often fits Dispatchers.Default.
Example: read text in IO, compute something in Default
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
import java.io.File
suspend fun countWords(file: File): Int {
val text = withContext(Dispatchers.IO) { file.readText() }
return withContext(Dispatchers.Default) {
text.split(Regex("\\s+")).count { it.isNotBlank() }
}
}
This example is intentionally short and not ideal in terms of performance (regexes and split can be expensive), but it shows the architectural boundary: reading is separate, processing is separate. And that makes the program more predictable.
7. Building reports: parallel and safe
Now let’s formalize the “heart” of our lecture — a function you can safely call from main. We’ll make it suspend so it works in the coroutine world without runBlocking inside (this is important discipline).
We’ll also add a parallelism parameter so you can adjust the concurrency limit.
Base variant: buildReportsParallel(...)
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.async
import kotlinx.coroutines.awaitAll
import kotlinx.coroutines.coroutineScope
import java.io.File
suspend fun buildReportsParallel(
files: List<File>,
parallelism: Int
): List<FileReport> = coroutineScope {
val io = Dispatchers.IO.limitedParallelism(parallelism)
files
.map { f -> async(io) { buildReport(f) } }
.awaitAll()
}
There’s an important point here: we use coroutineScope { ... }. This means all created async are “children” of the current coroutine, and the function won’t complete until all children complete. That’s structured concurrency in action: tasks are tied to a lifecycle, rather than “flying off into space and being forgotten.” Kotlin describes context and task hierarchy as the foundation of structured concurrency.
Usage from main
import kotlinx.coroutines.runBlocking
import java.io.File
fun main() = runBlocking {
val files = listOf(File("a.txt"), File("b.txt"), File("c.txt"))
val reports = buildReportsParallel(files, parallelism = 2)
for (r in reports) {
println("${r.file.name}: ${r.lineCount} lines") // for example: a.txt: 10 lines
}
}
If one file is “bad”: how not to bring everything down at once
Now for an important (and slightly philosophical) part: concurrency isn’t only about “faster.” It’s also about “what if one of them fails?”
The default behavior of such code is fairly strict: if an exception happens inside one async, it will “surface” on await()/awaitAll(). That is, the error becomes visible where you wait for the result. And that’s honestly a good model: you either got all reports, or you learned that it didn’t work.
Within this lecture, we won’t build a complex partial-results strategy (that’s a separate big topic), but we’ll show a simple technique: if you want processing of other files to continue, you can catch the error inside async and return null.
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.async
import kotlinx.coroutines.awaitAll
import kotlinx.coroutines.coroutineScope
import java.io.File
suspend fun buildReportsOrSkip(
files: List<File>,
parallelism: Int
): List<FileReport> = coroutineScope {
val io = Dispatchers.IO.limitedParallelism(parallelism)
files
.map { f ->
async(io) {
try { buildReport(f) } catch (e: Exception) { null }
}
}
.awaitAll()
.filterNotNull()
}
This isn’t “perfect error handling”, but as a learning stepping stone it’s very useful: you see how errors affect result collection and why awaitAll() is the point where the result becomes “mandatory.”
8. Common mistakes in parallel file processing
Mistake No. 1: running file operations on Dispatchers.Default.
It looks harmless: “well, it works.” But this way you start blocking threads that are intended for computations. As a result, another coroutine that honestly computes something CPU-heavy suddenly has to wait because you took its thread to read a file. The remedy is simple and boring, like safety rules: always send file operations to Dispatchers.IO.
Mistake No. 2: creating async inside map, but forgetting about await/awaitAll.
This is the classic “I started everything, but nothing happened.” In fact, it did happen — you created a batch of Deferred, but never waited for the results. In the best case, the program finishes early; in the worst case, you get weird effects because work is still running while you’re already printing “done.” It’s fixed by discipline: if you used async, then somewhere there must be await() or awaitAll().
Mistake No. 3: starting too many tasks at once without a limit.
When there are many files, the temptation is huge: “coroutines are lightweight.” Yes, but the disk isn’t made of rubber. If you start thousands of read tasks at once, you create contention for a resource that is already basically one (okay, two if you have two disks). Often this slows execution down and makes the system less responsive. A good habit is to use limitedParallelism(n) and start with a small n, for example 4.
Mistake No. 4: doing async(Dispatchers.IO) { withContext(Dispatchers.IO) { ... } }.
This looks like “double protection against mistakes”, but in practice it’s just extra nesting and noise. You either start the coroutine on the IO dispatcher, or switch to IO inside. In today’s style it’s more convenient to do async(ioLimited) { ... }, and inside just call the file-work code (or functions that switch internally — but then don’t do it a second time).
Mistake No. 5: mixing file reading and heavy processing in a single IO context.
This kind of code often appears by itself: “I’m already in IO, so I’ll do everything here.” But the IO context is intended for blocking operations, not for “spend a long time crunching regexes and parsing a gigabyte of text.” As a result, you clog the I/O pool with computations and lose predictability. Much cleaner: I/O separately, CPU separately — at least in two steps.
GO TO FULL VERSION