1. Text analysis pipeline
When you hear “text analysis”, it’s easy to imagine something scary: neural networks, linguistics, and an instructor who says “this is obvious”. In reality, in a simple educational Text Analyzer, almost everything rests on a very down-to-earth idea: we consistently transform one data type into another until we get a convenient result. That’s the pipeline: several steps, each with a clear input and output.
Imagine you’re given a large text and asked: “Which words occur most often?” If you code it “head-on”, you can drown in loops, temporary variables, and checks. The pipeline saves you because you do exactly 5–6 standard steps: normalization → tokens → frequencies → sorting → top‑N → formatting. And at each step, you get a structure that’s pleasant to work with.
Here’s the diagram (very technical, but honest):
flowchart TD
A["Raw text (String)"] --> B["Normalized text (String)"]
B --> C["Tokens (List⟨String⟩)"]
C --> D["Frequencies (Map⟨String, Int⟩)"]
D --> E["Sorted elements (List⟨Entry⟩)"]
E --> F["Top-N (List⟨Pair⟨String, Int⟩⟩)"]
F --> G["Report (String)"]
The key idea: we don’t “try to solve everything at once”; each time we make a small transition between understandable types.
Type map across the pipeline steps
So the pipeline doesn’t turn into magic, it’s useful to keep a “type map” in mind ahead of time: what we had and what it became. Then the code reads like a route in a navigator rather than a quest of “find where the bracket disappeared”.
Below is a practical table you can come back to when you catch yourself thinking, “Wait, why is there entries here and not map?”
| Step | What we do | Input | Output | Example operations |
|---|---|---|---|---|
|
Normalize the text | |
|
|
|
Tokenize | |
|
|
|
Count frequencies | |
|
|
|
Select top‑N | |
List<Map.Entry<String, Int>> or List<Pair<String, Int>> | |
|
Format | list of Pair/Entry | |
|
Important: for now we’re building a simple analyzer. We’re not diving into “smart tokenization”, we’re not accounting for every punctuation mark in the world, and we’re not deciding the fate of the hyphen in a word like “mother‑of‑a‑programmer”. We’re building a reliable baseline pipeline that can be improved later.
2. Text normalization
Text normalization is the moment when you turn user input into something suitable for honest processing. A user can paste text with line breaks, multiple spaces, tabs, and random uppercase letters. If we start split() directly on this “raw” input, we’ll get garbage: empty tokens, “Kotlin” separately from “kotlin”, and weird frequency results.
Mini-normalization: trim() and lowercase()
Let’s start with the basics. We’ll remove whitespace at the edges and convert everything to lowercase. This immediately reduces “noise” a lot.
fun main() {
val raw = " Kotlin KOTLIN kotlin "
val normalized = raw.trim().lowercase()
println(normalized) // kotlin kotlin kotlin
}
The idea here is simple: if case isn’t important for meaning (and in our analyzer it isn’t), it’s better to normalize case right away.
Normalizing whitespace via Regex("""\s+""").replace(...)
Now for a beginner pain point: “Why do I get empty strings after split(" ")?” Usually because the text contains lots of spaces, line breaks, and tabs. We want to “collapse” any sequence of whitespace characters into a single regular space.
import kotlin.text.Regex
fun main() {
val raw = " Kotlin Kotlin \n is\tgreat "
val spaces = Regex("""\s+""")
val normalized = spaces.replace(raw.trim().lowercase(), " ")
println(normalized) // kotlin kotlin is great
}
Note "\s+". That means “one or more whitespace characters”. In other words, we convert spaces, "\n", and "\t" into one uniform form. After that, tokenization becomes predictable.
If the text ends up empty after normalization, it’s convenient to pick a fallback scenario right away using takeIf { ... } ?: ... . It’s that “take the value if it fits, otherwise default” style that reads linearly.
3. Tokenization
Tokenization sounds scary, but in our context it literally means “split the string into words”. The key is not to rush: if you do split(" ") on dirty text, you’ll get empty elements. If you already normalized whitespace, the task becomes almost boring — which is a great sign of correct code.
Simple tokenization after normalization
Assume we already have a string where there is exactly one space between words. Then split(" ") will work reliably.
fun main() {
val normalized = "kotlin kotlin is great"
val tokens = normalized.split(" ")
println(tokens) // [kotlin, kotlin, is, great]
}
Filtering empty tokens: filter { it.isNotBlank() }
Even if you try hard, “dirt” sometimes sneaks in. For example, the user entered only spaces, or your normalization doesn’t cover some case yet. That’s why filtering with isNotBlank() is a cheap and useful safety net.
fun main() {
val normalized = "kotlin kotlin " // intentionally "dirty"
val tokens = normalized
.split(" ")
.filter { it.isNotBlank() }
println(tokens) // [kotlin, kotlin]
}
This filter doesn’t do magic—it just throws away empty strings and strings made of spaces. And it’s exactly the kind of “small step” that makes the whole pipeline more robust.
4. Counting frequencies
This is where the pleasant part begins: we’re no longer wrestling with a string; we’re working with a normal collection List<String>. Our goal is to get a Map<String, Int>, where the key is the word and the value is how many times it occurred.
In the good old days (roughly yesterday), you might have built a MutableMap and incremented a counter by hand. That’s a useful skill, but Kotlin offers a more compact and readable option: groupingBy { ... }.eachCount().
fun main() {
val tokens = listOf("kotlin", "kotlin", "is", "great", "great")
val freq: Map<String, Int> = tokens.groupingBy { it }.eachCount()
println(freq) // {kotlin=2, is=1, great=2}
}
Why is this convenient as a pipeline step? Because the output is a structure that fits the “top‑N” task perfectly: frequencies are already counted; all that’s left is to select the largest ones.
5. Selecting top‑N
The naive idea sounds like: “I want to sort a Map.” And Kotlin will gently hint that a Map is not a list, and you can’t sort a “map” directly the same way you sort a list. But a Map has entries: a set of “key-value” pairs that you can turn into a list and sort however you like.
Top‑N via entries → sortedByDescending → take
fun main() {
val freq = mapOf("kotlin" to 2, "is" to 1, "great" to 2, "fun" to 5)
val top2 = freq.entries
.sortedByDescending { it.value }
.take(2)
for (e in top2) {
println("${e.key}: ${e.value}")
// fun: 5
// kotlin: 2 (or great: 2 — we don’t fix the order on ties)
}
}
There’s an important nuance for beginners: if values are equal (for example, kotlin=2 and great=2), the exact order between them usually doesn’t matter. Later you can add a secondary alphabetical sort, but in this lecture we keep the focus on the basic pipeline.
Converting Entry to Pair
Sometimes Entry is inconvenient for formatting. Then you can make one more small step: convert Entry into a Pair<String, Int>.
fun main() {
val freq = mapOf("kotlin" to 2, "is" to 1, "great" to 2)
val top = freq.entries
.sortedByDescending { it.value }
.take(2)
.map { it.key to it.value }
println(top) // [(kotlin, 2), (great, 2)] (or the other way around on ties)
}
Notice how this matches the pipeline philosophy: one more simple data transformation to make the next step easier.
6. Formatting the report
Up to this point, we were “honestly counting”, and that was the computational part. Now we move to presenting the result to the user: we want a nice report string that we can print with println. The key is not to start mixing counting logic with formatting logic. Even if you do everything in one main, keep the mental separation: “here I count, here I display”.
Printing with joinToString("\n")
fun main() {
val top = listOf("fun" to 5, "kotlin" to 2, "is" to 1)
val report = top.joinToString(separator = "\n") { (word, count) ->
"$word: $count"
}
println(report)
// fun: 5
// kotlin: 2
// is: 1
}
Destructuring (word, count) works here because Pair supports component functions. The result is compact and readable.
A bit of alignment via padEnd and padStart
We’ve already covered padStart/padEnd for strings, so we can make the report slightly neater so the numbers line up in a column.
fun main() {
val top = listOf("fun" to 5, "kotlin" to 12, "is" to 1)
val report = top.joinToString(separator = "\n") { (word, count) ->
"${word.padEnd(10)} | ${count.toString().padStart(3)}"
}
println(report)
// fun | 5
// kotlin | 12
// is | 1
}
This still isn’t a “big formatter”, just a pleasant look for the console.
7. Building a mini Text Analyzer app
Now we’ll do what the whole pipeline was for: connect the steps into a single scenario. The idea is: read text, normalize it, tokenize it, count frequencies, take top‑N, and print the report. For now we won’t extract steps into separate functions—that will be the next lecture. Here it’s more important to feel the linearity of the transformations and see how the types “move” from step to step.
Variant with intermediate vals
import kotlin.text.Regex
fun main() {
println("Enter text:")
val rawText = readln()
val normalized = Regex("""\s+""").replace(rawText.trim().lowercase(), " ")
val tokens = normalized.split(" ").filter { it.isNotBlank() }
val freq = tokens.groupingBy { it }.eachCount()
val topN = freq.entries
.sortedByDescending { it.value }
.take(5)
.map { it.key to it.value }
val report = topN.joinToString("\n") { (w, c) -> "$w: $c" }
println(report)
}
This code is “a bit longer”, but it honestly shows the thought process: a string becomes a list, the list becomes a map, the map becomes top‑N, top‑N becomes text. For early attempts, that’s better than a “super chain in 1 line”.
Chained variant with scope functions
If you’re already familiar with scope functions from the previous lecture, you can assemble the result in a more “pipeline-like” style. Here also is convenient for inserting debug printing without breaking the main result (it returns the original object).
import kotlin.text.Regex
fun main() {
val report = readln()
.let { Regex("""\s+""").replace(it.trim().lowercase(), " ") }
.also { println("Normalized: $it") } // Normalized: ...
.split(" ")
.filter { it.isNotBlank() }
.groupingBy { it }.eachCount()
.entries
.sortedByDescending { it.value }
.take(5)
.joinToString("\n") { "${it.key}: ${it.value}" }
println(report)
}
Don’t try to write like this all the time. It’s a style that works well when the chain is short and readable. If you start getting confused about where you have a String, where you have a List, and where you have a Map, it’s better to return to intermediate vals—this isn’t “losing”, it’s normal engineering caution.
8. Common mistakes when building a Text Analyzer pipeline
Mistake #1: tokenization before normalization.
If you start with split(" ") on raw text, you’re almost guaranteed to get empty tokens and weird results. The problem isn’t split; it’s that spaces can be multiple, plus there are "\n" and "\t". First reduce whitespace characters to a single space, and only then split into words.
Mistake #2: you forgot isNotBlank() filtering and got surprised by empty words.
Even after normalization, edge cases sometimes remain: empty input, spaces at the edges, unexpected combinations. If you don’t drop empty tokens, you can end up with the “word” "", and it will suddenly become the most frequent (because emptiness is very popular... philosophers might approve, users won’t).
Mistake #3: trying to “sort a Map directly”.
Map is a “key → value” structure, and sorting is a story about ordered collections. So the correct path is almost always: freq.entries → sortedByDescending { it.value } → take(n). Once you accept this as normal, top‑N stops being a mystery.
Mistake #4: mixing counting and formatting into one mess.
When you count frequencies and at the same time build a multi-line string right in the middle, the code quickly becomes unreadable: it’s unclear where the logic ends and where the “pretty output” begins. Even if you write everything in one main, keep formatting as a separate final step (for example, a separate variable report).
Mistake #5: a chain that’s too clever, where you lost the types.
Scope functions and operation chains are great, but they shouldn’t turn into a magic trick with a disappearing rabbit. If in the middle of the chain you’re no longer sure what you’re holding (String? List<String>? Map<String, Int>?), stop, create intermediate vals, and give them sensible names. This doesn’t slow a programmer down—it speeds up debugging.
GO TO FULL VERSION