Concepts / Reading and Processing Files

Reading and Processing Files

The same input can support different analyses. Counting complete email addresses measures individual senders. Counting domains combines senders from the same organization.

  • Programming

From Raw Messages to Useful Questions

A mail log may contain thousands of messages, but the raw lines do not immediately answer useful questions. You may want to know which sender appears most often or which organization sends the most mail. The key move is to transform repeated values into a structured summary. In Python, a dictionary can store that summary as a histogram: each key represents a unique value, and its value records how many times that key appears.

Key point

The same input can support different analyses. Counting complete email addresses measures individual senders. Counting domains combines senders from the same organization.

Adding Each Sender to a Histogram

Suppose each processed line gives you one sender email address. The dictionary key is that address. When the address appears for the first time, its count should become 1. When it appears again, the existing count should increase by 1. The dictionary groups identical values automatically because all occurrences of the same sender use the same key.

a@example.orgfirst messagea@example.org1b@example.orgfirst messageb@example.org1a@example.organother messagea@example.org2create keycreate keyincrease existing value
How does each email address become a dictionary key, and how does its count change when another message from that sender is read?
python

The expression counts.get(email, 0) looks up the current value for email. If the key is not present, it supplies 0 instead. Adding 1 therefore handles both cases in one operation: a new sender receives a count of 1, while an existing sender receives its previous count plus 1.

Worked example

Three Messages, Two Senders

Process the sender sequence a@example.org, b@example.org, a@example.org.

  1. 1
    First sender

    The key a@example.org does not yet exist, so its count becomes 0 + 1, or 1.

  2. 2
    Second sender

    The key b@example.org does not yet exist, so it receives a count of 1.

  3. 3
    Repeated sender

    The key a@example.org already has the value 1, so its count becomes 1 + 1, or 2.

Result

The histogram is {"a@example.org": 2, "b@example.org": 1}.

Tracing the Maximum Sender

Once the histogram is complete, finding the most prolific sender requires a second pass through the dictionary. The loop must track two pieces of information together: the largest count seen so far and the sender associated with that count. For each dictionary entry, compare its count with the current maximum. If it is larger, replace both tracked values.

startlargest = 0a@example.orgcount 2a@example.orglargest = 2b@example.orgcount 1a@example.orglargest = 2resulta@example.orginspect2 > 0inspect next1 is not greater than 2report winner
How does the loop compare each sender's count and update the current maximum and winning sender?
python

The comparison uses a strict greater-than test. A sender replaces the current winner only when its count is larger than the stored maximum. The loop therefore preserves the first maximum it encounters if another entry has an equal count. The important design idea is not just finding a large number; it is keeping the number and the key that produced it together.

What do you think happens?

If the histogram contains a@example.org with 2 messages and b@example.org with 1 message, what will winner contain after the loop?

  • a@example.org

  • b@example.org

  • 2

  • None

Reveal answer
Answer:

a@example.org

The loop first records a@example.org because 2 is greater than the initial maximum of 0. The later count of 1 does not replace it.

Moving from Senders to Organizations

A full email address gives a fine-grained view: different addresses are different keys. To study organizations instead, extract the domain portion and use that domain as the histogram key. The domain is the part after the @ symbol. Splitting the address at @ and selecting the second part changes what counts as the same group.

file linesender dataemail addressa@example.orgsender histogramkey and countdomainexample.orgdomain histogramorganization countread and extractuse full address as keysplit at @use domain as key
How does data move from each input file line to the extracted email address and then into the frequency histogram?
emaila@example.orgdomainexample.orgsplit@[1]second partemail.splitselectreturn
How does splitting an address at the @ symbol transform a full email address into its domain part?
python

The split method separates the address at the @ character. Index 1 selects the second resulting part, which is the domain in this pattern. The source material also describes an alternative using find to locate @ and slicing from the position after it; split is presented as the cleaner approach.

Comparing Two Levels of Detail

a@example.org2 messagestwo keys2 and 1example.org3 messagesone key3b@example.org1 messagekeep complete addresskeep complete addresscombine shared domainextract domainextract domain
How do the histogram keys and frequency totals change when messages are grouped by complete email address versus shared domain?
Analysis keyWhat one key representsResult for a@example.org and b@example.org
Full email addressOne individual senderTwo separate keys
DomainAn organization or domainOne shared key if both domains match

Changing the key changes the grouping and therefore changes the question answered by the histogram.

Worked example

Same Messages, Different Question

Analyze messages from a@example.org, b@example.org, and a@example.org first by complete address and then by domain.

  1. 1
    Count complete addresses

    The sender histogram gives a@example.org a count of 2 and b@example.org a count of 1.

  2. 2
    Extract domains

    Both addresses produce the domain example.org when split at the @ symbol and the second part is selected.

  3. 3
    Count domains

    All three messages accumulate under the single domain key example.org.

Result

The address analysis identifies a@example.org as the more frequent individual sender, while the domain analysis gives example.org a total of 3 messages.

python
Key point

The counting pattern stays the same. Only the value used as the key changes: use email for individual-sender analysis, or use email.split('@')[1] for domain analysis.

Mistakes That Distort the Count

  • Common mistake

    Using the full email address when the question asks for organization totals

    What it looks like
    counts[email] = counts.get(email, 0) + 1
    Why it fails

    Different people from the same domain remain separate keys, so their messages are not combined.

    Do this instead

    Extract the domain first and use domain as the dictionary key.

  • Common mistake

    Forgetting to add 1 to the existing count

    What it looks like
    counts[email] = counts.get(email, 0)
    Why it fails

    A repeated sender does not accumulate additional messages.

    Do this instead
    Use counts[email] = counts.get(email, 0) + 1.
  • Common mistake

    Tracking only the largest number during the maximum loop

    What it looks like
    largest = count
    Why it fails

    The final number does not identify which sender produced it.

    Do this instead

    Track both the maximum count and the corresponding key.

  • Common mistake

    Selecting the wrong result after splitting

    What it looks like
    domain = email.split('@')[0]
    Why it fails

    The first part is before the @ symbol, not the domain part described in this analysis.

    Do this instead

    Use email.split('@')[1] to select the second part.

Practice the Processing Pattern

MEDIUM

You have processed messages from alice@north.example, bob@south.example, alice@north.example, and carol@north.example. Describe the sender histogram and the domain histogram. Then identify the most frequent individual sender and the most frequent domain.

Hints
  • Use the complete address as the key for the first histogram.
  • Use email.split('@')[1] as the key for the second histogram.
  • For the maximum, compare each count while tracking both the winning key and its count.
Worked example

Checking the Practice Result

Use the four practice addresses to determine the two histograms and their maximum entries.

  1. 1
    Sender histogram

    alice@north.example appears twice. bob@south.example and carol@north.example each appear once.

  2. 2
    Domain histogram

    The north.example domain appears for Alice twice and Carol once, for a total of 3. The south.example domain appears once.

  3. 3
    Maximum entries

    The most frequent individual sender is alice@north.example with 2. The most frequent domain is north.example with 3.

Result

Sender counts: alice@north.example = 2, bob@south.example = 1, carol@north.example = 1. Domain counts: north.example = 3, south.example = 1.

The Processing Pipeline

  1. 1
    Open the mail log and process each line.
  2. 2
    Extract the sender email address from the line.
  3. 3
    Use the address as a dictionary key and increment its count with counts.get(key, 0) + 1.
  4. 4
    Iterate through the completed dictionary while tracking both the largest count and its key.
  5. 5
    If the question concerns organizations rather than individual senders, split each email at @ and count the resulting domain.
  1. 1
    A histogram is a dictionary in which keys identify values and dictionary values store their frequencies.
  2. 2
    The pattern counts[key] = counts.get(key, 0) + 1 handles both first appearances and repeated appearances.
  3. 3
    Finding a maximum requires tracking the winning key as well as its count.
  4. 4
    email.split('@')[1] changes the key from a complete address to a domain.
  5. 5
    Changing the key changes the granularity and meaning of the analysis.

Key Takeaways

  • Use a dictionary histogram to convert repeated email addresses into frequency counts.
  • Increment counts safely with counts[key] = counts.get(key, 0) + 1.
  • Find the most frequent entry by tracking both the largest count and its associated key.
  • Extract a domain with email.split('@')[1] when the analysis should group messages by organization.
  • The choice of dictionary key determines whether the result describes individual senders or broader domains.

Related Topics

Related concepts

More concepts starting with R