Practical walkthrough: word frequencies

1. Introduction

The task “count how many times each word occurs” looks like something from a school programming contest, where people code for speed and feed off the energy of the Enter key. But in real programs it shows up all the time: comment analysis, popular query search, log processing, building simple text reports, and even tracking frequent causes in task descriptions.

We’ll build a mini text analyzer that takes a string, brings it into a normal form, splits it into words, and builds a frequency map.

One important idea right away: frequencies are almost never counted “just because”. Before counting, data usually needs to be brought to a single standard; otherwise, the same word splits into several keys. For example, "Kotlin", "kotlin", and " kotlin " are three different strings, but semantically they are the same word. That’s why we’ll end up with a clear processing pipeline.

Here is today’s “route map” for the lecture:

flowchart LR
    A[Raw text] --> B[Normalization]
    B --> C[Tokens: words]
    C --> D["Frequencies: Map⟨String, Int⟩"]
    D --> E[Sorting and Output]

2. Step 1: text normalization

Normalization is the moment when we stop being romantics and start being engineers. User text can be anything: extra spaces, different casing, random line breaks, sometimes even “KoTlIn”. If we don’t align the input, the frequency map becomes meaningless: different spellings become different keys.

The most basic minimum normalization we already know: trim() and lowercase(). This is often enough to remove “junk” at the edges and bring words to a single case.

Mini example: trim() + lowercase()

fun main() {
    val raw = "  Kotlin  kotlin  KOTLIN  "
    val normalized = raw.trim().lowercase()

    println(normalized) // kotlin  kotlin  kotlin
}

Note: we haven’t solved the problem of double spaces in the middle of the string yet. That will be the next part of the story: during tokenization we’ll filter out empty pieces anyway.

Punctuation normalization (simplified)

In an ideal world, we’d throw away punctuation so that "kotlin," and "kotlin" become one word. But today we’re not doing complex tokenization and we’re not using regular expressions. Instead, we can do a small “practical” trick: replace a few of the most common characters with a space.

fun main() {
    val raw = "Kotlin, kotlin! Kotlin?"
    val normalized = raw
        .lowercase()
        .replace(",", " ")
        .replace("!", " ")
        .replace("?", " ")

    println(normalized) // kotlin  kotlin  kotlin
}

This isn’t perfect (for example, hyphens and quotes will remain), but it already gives much more “honest” keys. And importantly: we normalize before counting frequencies so the keys are correct from the start.

3. Step 2: tokenization

Tokenization is a scary word that in practice means “split text into parts”. Today, tokens are words, and we’ll get them via split. The main pitfall: if there are several spaces in a row, then split(" ") can produce empty strings "". And an empty string is still a string, and it will calmly become a key in a Map if we don’t throw it away.

So our standard technique: after split we necessarily do filter { it.isNotBlank() }. It’s like a “sieve”: everything empty and consisting of spaces flies into the trash.

Mini example: split and filtering empty tokens

fun main() {
    val normalized = "kotlin  kotlin   is   great"
    val tokens = normalized
        .split(" ")
        .filter { it.isNotBlank() }

    println(tokens) // [kotlin, kotlin, is, great]
}

A typical trap: an empty-string key appears

To feel the problem hands-on, it’s useful to see what happens without the filter:

fun main() {
    val normalized = "a  b"
    val tokens = normalized.split(" ")

    println(tokens) // [a, , b]
}

That blank spot in the middle is "". If you then count frequencies “as is”, you’ll get the key "" with frequency 1, and then you’ll stare at the report approximately like a cat staring at a cucumber: it exists, but why — unclear.

4. Step 3: frequencies via groupingBy { ... }.eachCount()

Now comes the most pleasant part: once the preparation is done, counting frequencies turns into one readable line. We take a list of tokens and say: “group by the word itself and count how many are in each group”. For this, groupingBy { it }.eachCount() is perfect.

In type terms it looks like this: we have a List<String>, and the output will be a Map<String, Int>. This is the frequency map: word → how many times it occurred. In Kotlin, the standard library directly supports this scenario: groupingBy returns a special Grouping object, to which you can apply the eachCount() aggregation.

Mini example: word frequencies

fun main() {
    val tokens = listOf("kotlin", "kotlin", "is", "great")
    val freq = tokens.groupingBy { it }.eachCount()

    println(freq) // {kotlin=2, is=1, great=1}
}

Note about the key: normalization inside keySelector

If we’re working not with already-normalized tokens, but, for example, with “raw” words, it’s better to normalize right inside the keySelector. This reduces the risk of forgetting the processing somewhere in the middle of the program.

fun main() {
    val rawWords = listOf("Tea", " tea ", "TEA", "Coffee")
    val freq = rawWords.groupingBy { it.trim().lowercase() }.eachCount()

    println(freq) // {tea=3, coffee=1}
}

5. Step 4: sorting and output

Almost always, the Map we got is not a report yet, but “raw material for a report”. The user wants to see the result in some order: for example, the most frequent words at the top. But a Map has no obligation to preserve the order we want for printing. So we usually take entries, sort them, and only then print.

Structurally, it’s convenient to read it like this: freq.entries is a set of “key-value” pairs, i.e., “word-frequency”. It’s easy to turn into a list and sort, and then output line by line.

Sorting by descending frequency

fun main() {
    val freq = mapOf("kotlin" to 5, "java" to 2, "go" to 3)

    val sorted = freq.entries.sortedByDescending { it.value }
    for (e in sorted) {
        println("${e.key} -> ${e.value}")
        // kotlin -> 5
        // go -> 3
        // java -> 2
    }
}

Top‑N: show only the most frequent words

fun main() {
    val freq = mapOf("kotlin" to 5, "java" to 2, "go" to 3, "swift" to 1)

    val top2 = freq.entries
        .sortedByDescending { it.value }
        .take(2)

    for ((word, count) in top2) {
        println("$word -> $count")
        // kotlin -> 5
        // go -> 3
    }
}

Here we use destructuring (word, count) in the loop so the code reads “like a human”, rather than via e.key and e.value.

A bit of output alignment

We already know simple string operations, so we can make the output a bit neater. For example, align words to the width of the longest word. This is a small step toward making the program look like a tool rather than a draft.

fun main() {
    val freq = mapOf("kotlin" to 5, "java" to 2, "go" to 3)
    val width = freq.keys.maxOf { it.length }

    for ((word, count) in freq.entries.sortedByDescending { it.value }) {
        val left = word.padEnd(width)
        println("$left : $count")
        // kotlin : 5
        // go     : 3
        // java   : 2
    }
}

6. Mini app: Text Analyzer

Now we’ll do something that’s especially satisfying in programming: assemble a complete scenario from small operations that you can run. We won’t turn this into a big “combine harvester” — on the contrary, we’ll make a few small functions so each one is responsible for a clear step. This makes the code easier to read, eyeball-test, and debug.

Imagine that in our learning project we already have a console command menu, and we add a text analysis command. Even if the menu is still simple, the text-analysis logic itself will be a useful module: you can apply it to expense descriptions, notes, reviews — anything with text.

Text normalization function

Let’s make a function that brings text to a “more-or-less stable” form. We don’t promise to handle every symbol in the world, but for a learning example this is enough.

fun normalizeText(raw: String): String {
    return raw
        .trim()
        .lowercase()
        .replace(",", " ")
        .replace(".", " ")
        .replace("!", " ")
        .replace("?", " ")
}

If you want to add more characters, add them, but it’s important to keep the principle: normalization must happen before tokenization.

Tokenization function: string → list of words

Now let’s make a function that takes normalized text and turns it into a list of words. The key point is filtering with isNotBlank().

fun tokenize(text: String): List<String> {
    return text
        .split(" ")
        .filter { it.isNotBlank() }
}

If in the future you want to improve tokenization, you’ll have exactly one place to change.

Frequency counting function

Now the shortest part: take a List<String> and get a Map<String, Int>. This is our “statistics engine”. eachCount() gives us a frequency map without storing lists of groups — efficient and semantically correct.

fun countFrequencies(tokens: List<String>): Map<String, Int> {
    return tokens.groupingBy { it }.eachCount()
}

Build a report string

Printing “head-on” via many println calls works, but often it’s nicer to assemble the report text first and then output it in one chunk. It enforces discipline: computation separately, output separately. And yes, StringBuilder for long text is friendly to both performance and your nervous system.

import kotlin.text.StringBuilder

fun buildReport(freq: Map<String, Int>, top: Int): String {
    val sb = StringBuilder()
    val sorted = freq.entries.sortedByDescending { it.value }.take(top)

    for ((word, count) in sorted) {
        sb.append(word).append(" -> ").append(count).append('\n')
    }
    return sb.toString()
}

Note: it’s only a few lines, but we already got a function you can reuse a hundred times.

Connect everything in analyzeText(...)

Now let’s make a turnkey function: it takes raw text, runs it through the pipeline, and returns a report.

fun analyzeText(raw: String, top: Int): String {
    val normalized = normalizeText(raw)
    val tokens = tokenize(normalized)
    val freq = countFrequencies(tokens)

    return buildReport(freq, top)
}

This is a direct illustration of the principle “normalization → tokens → frequencies → output”, just in code.

main: read a line and print the result

Let’s assemble a minimal run. The user enters text on a single line, and we print the top 10 words. If there are fewer words, it prints as many as there are; take(top) doesn’t complain.

fun main() {
    println("Enter text in one line:")
    val raw = readln()

    val report = analyzeText(raw, top = 10)
    println("Top words:")
    print(report)
}

If you run it and enter, for example:

Kotlin, kotlin! Is great. Kotlin is pragmatic.

then the output will be approximately like this (the order for equal frequencies may differ):

Top words:
kotlin -> 3
is -> 2
great -> 1
pragmatic -> 1

Bringing it closer to reality: analyze a list of texts

Sometimes the data is not one string, but many strings. For example, we have a list of “expense notes” (the user wrote comments like “coffee and bun”, “taxi home”, “coffee in the office”). Even if in our project expenses are still represented by simple structures, we can take a list of strings, combine it into one overall text, and then count frequencies.

Important: we don’t have to know “smart” collection operations to do this. You can safely build the text with a StringBuilder in a loop — that’s already familiar and transparent.

import kotlin.text.StringBuilder

fun joinTexts(lines: List<String>): String {
    val sb = StringBuilder()
    for (line in lines) {
        sb.append(line).append(' ')
    }
    return sb.toString()
}

And now apply the analysis:

fun main() {
    val notes = listOf(
        "Coffee and sandwich",
        "coffee to go",
        "Taxi home"
    )

    val raw = joinTexts(notes)
    val report = analyzeText(raw, top = 5)

    println("Частые слова в заметках:")
    print(report)
}

You’ll get a useful mini-report: which words appear most often in the comments. Yes, this isn’t “corporate-level business analytics” yet, but it’s honest automation: the computer does the boring work, the human reads the output and thinks.

7. Common mistakes

Mistake #1: count frequencies using “dirty” keys and be surprised by the result.
If you don’t apply trim() and lowercase() before counting, you’ll quickly get a situation where "Tea", "tea", and " tea " are considered different words. The frequency map will look “correct” technically, but useless semantically. So normalization is not decoration; it’s part of the key contract.

Mistake #2: forget filtering after split and accidentally count the word "".
split(" ") on text with double spaces creates empty tokens. If you don’t remove them with filter { it.isNotBlank() }, the empty string becomes a normal key in the frequency map. After that, you can spend a long time figuring out who this mysterious key is whose frequency is 17.

Mistake #3: mix computation and printing into one big “sheet”, and then be unable to debug.
When code does everything in one chain and prints immediately, any problem turns into a quest. It’s much easier to keep intermediate results in clear variables: normalized, tokens, freq, sorted. This style may feel “more verbose”, but in practice it saves time and nerves.

Mistake #4: print a Map directly and expect the order to be “the way you want”.
Even if in some version or environment a Map looks sorted, that is not a report contract. If you need an order, you must set it explicitly: take entries, sort by frequency, and only then print. Otherwise the program may “jump” in the output, and the user will think the frequencies changed, although only the display order changed.

Mistake #5: try to make “perfect tokenization” and drown in details.
It’s very easy to start adding handling for every possible character set, language, and exception — and lose the point of the lesson. In practice, it’s better to keep tokenization in a separate function and improve it gradually. Today it’s enough that it’s predictable, not perfect.