Decomposing a Text Analyzer: steps and assembly via let(::...)

1. A pipeline as an assembly line and step types

When you’re just putting together your first pipeline, there’s a pleasant feeling: “Look, mom, I’m a functional programmer.” But then two days pass, you open your code — and you get the effect of “I didn’t write this, my night-time doppelganger did.” The problem isn’t the pipeline; it’s that the brain has a hard time holding both the meaning of each step and the data type at each step at the same time.

Imagine you have a chain of 10 operations: trim, lowercase, replace, split, filter, groupingBy, eachCount, entries, sortedByDescending, take. You can write all of that in a single line, but one mistake turns debugging into the game “guess where we lost the meaning.” So we do what grown-up developers do: extract steps into separate functions with self-explanatory names.

When we say “decomposition,” it’s not just “split into functions because that’s how it’s done.” We want to get an assembly line where each machine has a clear input and a clear output. Then the line can be assembled in any way, tested in parts, and you can replace one machine without tearing down the whole factory.

To make it easier for the brain, it helps to lock in the result types at each step. Here’s what our text analyzer “assembly line” looks like:

normalizeText

String

String

tokenize

String

List<String>

countTokens

List<String>

Map<String, Int>

topN

Map<String, Int>

List<Pair<String, Int>>

formatTop

List<Pair<String, Int>>

String

If you noticed, almost everything “fits together” perfectly here: the output of one step is the input of the next. That’s exactly the case where let(::...) becomes especially readable, because let semantically means “pass the result forward” and returns the block result. This idea maps nicely to the idiom value?.let { ... } and to “transforming a value via a block” in Kotlin.

Function responsibility boundaries

It’s very tempting to make a function analyzeText() and shove everything in there: normalization, counting, formatting, and printing to the console. But then we get a “combine harvester” that’s hard to reuse: for example, you’ll want to output top-N not as text but as a table — and you’ll have to rewrite half of it.

A good responsibility boundary sounds boring, but works great: a function does one clear thing and returns a result, without printing it and without reading input. Printing and input are the domain of main() (or a separate CLI layer), while pure computation is the domain of the pipeline functions. Then the steps can be reused in another project, in tests, or with a different output mode.

2. Text Analyzer pipeline steps

Step 1 — normalizeText(text: String): String

Normalization is like “tidying the room before looking for socks.” If we immediately do split(" ") on raw text, we’ll get tokens like "" (empty string), "\n" or words in different cases. Normalization makes the input predictable: consistent spaces, consistent case, and no “extra” edges.

Example of a small function (note: 1 step — 1 meaning):

import kotlin.text.Regex

fun normalizeText(text: String): String =
    Regex("""\s+""")
        .replace(text.trim().lowercase(), " ")

Here we do three things: trim() removes whitespace at the edges, lowercase() brings everything to the same case, and Regex("""\s+""") collapses any “whitespace obstacle course” into a single space.

Step 2 — tokenize(text: String): List<String>

Tokenization is the moment when a string becomes a collection, and this is exactly where “invisible” bugs often appear: extra spaces produce empty elements, tabs behave differently, and a string of only spaces becomes a list with a single empty token. Our goal is to get a list of words where every word is non-empty.

A tokenization function in the simplest form can be like this:

fun tokenize(text: String): List<String> =
    text.split(" ")
        .filter { it.isNotBlank() }

Yes, after normalization we already have “one space,” but filter { it.isNotBlank() } is a seat belt. In programming, seat belts don’t look cool, but they save lives (your time and nerves).

Step 3 — countTokens(tokens: List<String>): Map<String, Int>

At this stage we do what the whole party is for: turn a list of words into frequencies. Previously, many people wrote this manually via MutableMap, but Kotlin has a standard and very readable way: groupingBy { it }.eachCount().

The function ends up being almost “self-documenting”:

fun countTokens(tokens: List<String>): Map<String, Int> =
    tokens.groupingBy { it }.eachCount()

Input: a list of words. Output: a map “word → how many times it occurred”. It’s convenient to keep this step separate: it doesn’t depend at all on where the words came from. Want to tokenize by spaces — great. Want to tokenize by commas — fine. Want to extract words from JSON — also fine. Frequencies are still counted the same way.

Step 4 — topN(freq: Map<String, Int>, n: Int): List<Pair<String, Int>>

Top-N is where it’s easy to get confused about types: a Map can’t be “sorted” directly like a list, because sorting is about order, while a Map is semantically about lookup by key. So we work via entries: first we turn it into a list of entries, sort by value, take take, and only then choose the result shape.

We intentionally return List<Pair<String, Int>> because it’s convenient for formatting: a list of pairs is easy to turn into lines, a table, whatever you want.

fun topN(freq: Map<String, Int>, n: Int): List<Pair<String, Int>> =
    freq.entries
        .sortedByDescending { it.value }
        .take(n)
        .map { it.key to it.value }

Notice the last line: map { it.key to it.value }. This turns a Map.Entry<String, Int> into a Pair<String, Int>. We don’t carry Entry further, because it’s an “internal” map type — the next step doesn’t need it.

Step 5 — formatTop(items: List<Pair<String, Int>>): String

Formatting is the “presentation” layer. Here we turn data into nice text. And an important point: formatting must not count, sort, or filter anything. If you suddenly start recalculating frequencies in formatting, it means the step boundaries have blurred.

Let’s make a simple format: one line — one word and its count:

fun formatTop(items: List<Pair<String, Int>>): String =
    items.joinToString(separator = "\n") { (word, count) ->
        "$word: $count"
    }

Destructuring (word, count) makes the code human-friendly. We already know destructuring from previous topics, and here it simply boosts readability.

5. Assembling the pipeline via let(::...)

Now for the best part: we have five small functions that can be read independently. All that’s left is to assemble a scenario out of them so that it looks like an assembly line, not like a “tangle of variables.” let is perfect for this because it’s exactly about transforming a value into a new value.

Let’s start simple: input text → report:

fun analyze(text: String, top: Int): String =
    text
        .let(::normalizeText)
        .let(::tokenize)
        .let(::countTokens)
        .let { freq -> topN(freq, n = top) }
        .let(::formatTop)

There’s one “special” step here: topN requires two arguments, so we can’t just write .let(::topN) — we need to supply n. We solve this with a lambda with the parameter name freq, so we don’t multiply mysterious its.

Notice how linearly this reads: take the text, normalize it, tokenize it, count it, take top-N, format it. It’s a “story,” not a “technical instruction.”

6. Using it in main() and output example

Now let’s make a small main() that uses our analyzer. Important: main() handles input/output, and analyze handles computations. That way we don’t mix responsibilities.

fun main() {
    println("Enter text for analysis:")
    val text = readln()

    val report = analyze(text, top = 5)

    println("Top-5 words:")
    println(report)
}

If you run it and enter:

Kotlin kotlin is great, and Kotlin is fun

then the output will be roughly like this (after your normalization and simple tokenization by spaces, punctuation may remain in words — that’s fine for the current version):

Top-5 words:
kotlin: 3
is: 2
great,: 1
and: 1
fun: 1

Yes, great, with a comma is not a bug; it’s an honest result of the fact that we’re not doing “smart” punctuation cleanup yet. The main thing is: architecturally, we’re ready to improve any step independently.

7. Pipeline diagram

When you write pipelines, it’s useful to sometimes draw “pipes.” It sounds childish, but it’s exactly how you quickly catch type errors and step “mismatches.”

flowchart TD
    A["String (raw)"] --> B["normalizeText(): String"]
    B --> C["tokenize(): List⟨String⟩"]
    C --> D["countTokens(): Map⟨String, Int⟩"]
    D --> E["topN(n): List⟨Pair⟨String, Int⟩⟩"]
    E --> F["formatTop(): String"]

If at some point you want to replace formatting with a “table with alignment,” you just change the last block without touching the rest of the diagram. That’s the practical point of decomposition: changes are local.

8. Readability: let(::step) instead of a chain of val

Beginners often fall into one of two extremes. Either they write everything as one giant chain, or they make 15 intermediate variables like text1, text2, list3, map4, and it turns into “archaeological digs”: what was list3 and how is it different from list2?

let(::normalizeText) is a compromise. We don’t lose linearity, but we have clear step names. Also, these steps are easy to reuse elsewhere: for example, you can call tokenize(normalizeText(text)) separately and use the tokens for another task.

And one more bonus: if the pipeline breaks, you can temporarily expand it into intermediate vals in just one place, then switch it back — the functions themselves don’t care either way.

9. Diagnostics via also between steps

Sometimes you want to see what you got after each step. Doing println() inside normalizeText() is a bad idea: the function stops being pure and starts “making noise” in every usage. But we can temporarily insert diagnostics into the pipeline using also, because also returns the original object and is intended for side effects.

For example:

fun analyzeDebug(text: String, top: Int): String =
    text
        .also { println("RAW: $it") }
        .let(::normalizeText)
        .also { println("NORMALIZED: $it") }
        .let(::tokenize)
        .also { println("TOKENS: $it") }
        .let(::countTokens)
        .also { println("FREQ: $it") }
        .let { freq -> topN(freq, n = top) }
        .let(::formatTop)

Yes, it’s longer — but it’s a “diagnostic mode” you can turn on/off. The key point is that the step functions themselves remain pure and reusable.

10. Common mistakes

Mistake #1: “A combine harvester instead of a pipeline.”
Often a student makes one function that normalizes, tokenizes, sorts, and prints. At first it seems convenient, but then any change becomes a risk of breaking everything at once. It’s much more stable to keep steps small: one step — one meaning, one input — one output.

Mistake #2: a blurry topN contract: returning an inconvenient type.
If topN returns List<Map.Entry<String, Int>>, it kind of works, but you carry along a type intended for the internal life of a Map. Formatting ends up with odd constructs, and readability drops. It’s better to choose a result type that fits the next step, and for formatting it’s usually more convenient to have List<Pair<String, Int>>.

Mistake #3: “double it” and losing readability in let.
When you start writing nested lets and leave it everywhere, in a minute it becomes unclear what exactly is stored in it right now: a string, a list, or a map. That’s the case where it’s better to explicitly name the parameter: let { freq -> ... } or temporarily flatten the chain into a couple of intermediate variables with normal names.

Mistake #4: side effects inside computational steps.
Sometimes people add println() for debugging inside normalizeText() or countTokens(), and then forget to remove it. As a result, the functions can’t be reused properly: they start printing something “by themselves.” If you want diagnostics, it’s better to insert it via also where you assemble the pipeline, rather than hiding it inside the steps.

Mistake #5: trying to “glue” steps with different signatures via ::function.
let(::tokenize) works because tokenize accepts exactly what the previous step returns. But topN also requires n, so trying to do .let(::topN) won’t work. That’s normal: just use a lambda .let { freq -> topN(freq, n = 5) } and don’t try to “beat the compiler with the power of thought.”