Concepts / Sorting Dictionary Output by Frequency

Sorting Dictionary Output by Frequency

The source notes that dictionary output may appear random because the keys are stored in hash order rather than in the order in which they were added.

  • Programming
Interactive lab

Try it: Counting Words with a Dictionary

The py4e histogram pattern: split each line into words, count them with counts[word] = counts.get(word, 0) + 1, then find the most common word with a max loop or by sorting (count, word) tuples in reverse.

How it works

  1. Optionally remove punctuation with str.translate and lower-case the line, then split() it into words.
  2. For each word: counts.get(word, 0) returns the current count or 0 for a new word; store it plus one (or the if word not in counts form).
  3. A dict remembers insertion order: a word's position is fixed the first time it is seen.
  4. Max loop: keep the word whose count is strictly larger than the best so far (the first word wins a tie).
  5. Or build [(count, word) …], sort with reverse=True (ties ordered by the word, reversed) and slice [:N].

Default run (25 steps): Start with an empty dict. The text has 3 lines. … [:3] keeps the top 3: [(5, 'the'), (2, 'sat'), (2, 'on')].

Simplified: Printable-ASCII text up to 320 characters, 8 lines and 60 words, typed in the lab instead of read from a file or URL.

Educational simulation

Loading the simulation…

From File Layers to Word Counts

Text in a file arrives in layers. The outer layer consists of lines, and each line contains individual words. Counting every word therefore requires two levels of processing: an outer loop that selects each line and an inner loop that processes the words inside that line. The dictionary records the result by using each word as a key and its frequency as the associated value.

Key point

The inner loop completes all of its iterations for one line before the outer loop moves to the next line. This is what allows the program to count every word in every line.

The Two-Level Loop Trace

Imagine an input file containing two lines: 'the cat sat' and 'the dog sat'. The outer loop first selects the line 'the cat sat'. The inner loop then processes 'the', 'cat', and 'sat' in sequence. Only after those three words have been processed does the outer loop select 'the dog sat'. The inner loop then processes 'the', 'dog', and 'sat'.

the cat satouter loop: line 1theinner iteration 1theinner iteration 1catinner iteration 2the dog satouter loop: line 2doginner iteration 2satinner iteration 3satinner iteration 3processesprocessesprocessesthen selectsprocessesprocessesprocesses
What happens as the outer loop selects each line and the inner loop processes every word inside it?

Building the Counts Dictionary

For each word, the program checks whether that word is already a key in the counts dictionary. If the word is not present, the program adds it with a value of 1. If the word is already present, the program increases its existing value by 1. The if-else structure is important because the first occurrence needs to create the key, while later occurrences need to update the value.

Code walkthrough
counts = {}

for line in fhand:
    words = line.split()
    for word in words:
        if word not in counts:
            counts[word] = 1
        else:
            counts[word] += 1
  1. 1
    The outer loop handles the file's line layer.
  2. 2
    The inner loop handles the word layer inside the current line.
  3. 3
    The membership check chooses between creating a new dictionary entry and updating an existing one.
  4. 4
    The complete pattern counts words across all lines processed by the file loop.
  • The dictionary starts with no word keys.
  • The outer loop receives one line at a time from the file object.
  • split() produces the words for the current line.
  • The inner loop processes each word in that list.
  • A first occurrence receives the value 1.
  • A later occurrence uses the existing value and increments it.
File linethe dog satWord listthe, dog, satthefirst occurrencetherepeated occurrence1initial count2updated countsplitselectscreates key withappears againincrements
How does each word move from a file line into a dictionary key, and how does its count change when the word appears again?

A Complete Count Trace

Worked example

Tracing Two Input Lines

Count the words in the two lines 'the cat sat' and 'the dog sat'.

  1. 1
    First line: the

    The word is not yet a key, so the dictionary receives the entry 'the': 1.

  2. 2
    First line: cat

    The word is new, so the dictionary receives 'cat': 1.

  3. 3
    First line: sat

    The word is new, so the dictionary receives 'sat': 1. After the first line, the dictionary is {'the': 1, 'cat': 1, 'sat': 1}.

  4. 4
    Second line: the

    The word is already a key, so its value increases from 1 to 2.

  5. 5
    Second line: dog

    The word is new, so the dictionary receives 'dog': 1.

  6. 6
    Second line: sat

    The word is already a key, so its value increases from 1 to 2.

Result

The final dictionary is {'the': 2, 'cat': 1, 'sat': 2, 'dog': 1}.

What do you think happens?

After the first line 'the cat sat' has been processed, what should the dictionary contain?

  • {'the': 1, 'cat': 1, 'sat': 1}

  • {'the': 2, 'cat': 1, 'sat': 1}

  • {'the': 1, 'cat': 1, 'sat': 2}

Reveal answer
Answer:

{'the': 1, 'cat': 1, 'sat': 1}

Each of the three words has appeared once. The repeated word cases have not occurred yet because the second line has not been processed.

Incrementing Counts Compactly

Expanded assignmentCompound assignmentMeaning
counts[word] = counts[word] + 1counts[word] += 1Read the current value, add 1, and store the result back
counts[word] = counts[word] - 1counts[word] -= 1Subtract from the current value and store the result
counts[word] = counts[word] * 1counts[word] *= 1Multiply the current value and store the result
counts[word] = counts[word] / 1counts[word] /= 1Divide the current value and store the result

The source identifies +=, -=, *=, and /= as compact assignment operators. The word-counting pattern uses += 1.

From Raw Output to Frequency View

After the nested loops finish, printing the dictionary shows each word as a key and its count as the value. The source notes that dictionary output may appear random because the keys are stored in hash order rather than in the order in which they were added. This does not mean the counting failed: the word-to-count entries are still the important result.

A frequency-oriented view is useful when the goal is to identify the most common words quickly. In the source's Romeo and Juliet example, 'is', 'the', and 'and' appear 3 times, 'sun' appears 2 times, and many other words appear once. The dictionary captures these frequencies, but its unsorted display does not make the most frequent entries easy to spot. Additional processing is therefore needed to present the results in a more readable frequency order.

Word-count pairsis: 3, the: 3, and: 3, sun:23 occurrencesis, the, and2 occurrencessun1 occurrencemany other wordsgroup by frequencygroup by frequencygroup by frequency
How does a collection of word-to-count pairs become easier to read when presented from highest frequency to lowest?

Control-Flow Mistakes

  • Common mistake

    Processing only one word from each line

    What it looks like

    The inner loop is missing or is placed outside the outer loop.

    Why it fails

    The file contains lines, and each line contains multiple words. Without the inner loop, the program does not process every word within each selected line.

    Do this instead

    Use an outer loop for lines and place the word-processing loop inside it so the inner loop completes for every line.

  • Common mistake

    Resetting the dictionary for every line

    What it looks like

    Creating a new empty counts dictionary inside the outer loop.

    Why it fails

    Earlier line counts would be discarded when the dictionary is recreated for the next line.

    Do this instead

    Keep one counts dictionary for the complete counting process so entries from earlier lines remain available when later words are processed.

  • Common mistake

    Assigning 1 every time a word appears

    What it looks like

    Using counts[word] = 1 even when the word is already present.

    Why it fails

    A repeated word would not accumulate its frequency; its existing count would be replaced by 1.

    Do this instead

    Check whether the word already exists. Add it with 1 when it is new, and increment its current value when it is repeated.

  • Common mistake

    Assuming an apparently random dictionary display means the counts are wrong

    What it looks like

    Expecting printed keys to appear in the same order as the words were read.

    Why it fails

    The source explains that dictionary output may appear random because keys are stored in hash order rather than insertion order.

    Do this instead

    Inspect the key-value pairs and use additional organization when you need a frequency-focused display.

Practice the Trace

MEDIUM

Trace the input lines 'red blue red' and 'blue green'. Write the dictionary after the first line, then write the final dictionary after the second line. For each word, identify whether the program creates a new key or increments an existing value.

Hints
  • Process all three words from the first line before moving to the second line.
  • The second occurrence of red should use the existing red entry.
  • The second line contains a repeated blue and a new green.

What do you think happens?

After processing both practice lines, which final dictionary is correct?

  • {'red': 2, 'blue': 2, 'green': 1}

  • {'red': 1, 'blue': 1, 'green': 1}

  • {'red': 2, 'blue': 1, 'green': 2}

Reveal answer
Answer:

{'red': 2, 'blue': 2, 'green': 1}

The first line gives red two occurrences and blue one occurrence. The second line adds one more blue and one green, producing red: 2, blue: 2, and green: 1.

What to Remember

  1. 1
    The outer loop processes one file line at a time, while the inner loop processes every word in the selected line.
  2. 2
    A dictionary stores each word as a key and its frequency as the value.
  3. 3
    A new word receives a count of 1; a repeated word increments its existing count.
  4. 4
    counts[word] += 1 is a compact form of counts[word] = counts[word] + 1.
  5. 5
    The dictionary contains the frequency data, but additional organization can make the most frequent words easier to identify.

At a glance

  • Nested loops navigate the two layers of a text file: lines first, then words within each line.
  • The dictionary's keys are words and its values are occurrence counts.
  • The if-else structure distinguishes a word's first occurrence from later occurrences.
  • The += operator updates an existing count concisely without changing the operation.
  • Sorting or organizing the completed dictionary output helps reveal which words occur most frequently.

Related Topics

Related concepts

More concepts starting with S