String Methods: lower() and upper()

Python treats uppercase and lowercase letters as distinct characters.

Programming

Interactive lab

Try it: String methods, character by character

How Python's str methods (split, strip, replace, find, count, startswith, join, in …) scan a string character by character — and that each call returns a new value while the original string never changes.

How it works

Each method call reads the string it is called on from left to right (strip also trims from the right end).
split() with no separator skips runs of whitespace and never returns empty strings; split(sep) cuts at every occurrence of sep and can return empty pieces; maxsplit limits the number of cuts.
find, index, count, replace and `in` slide a window along the string; occurrences never overlap, find gives -1 and index raises ValueError when nothing matches.
strip(chars) removes any of the given characters (a set, not a substring) from the ends until it meets one that is not in the set.
Strings are immutable: every call returns a new str, list, int or bool, and the next call in the chain works on that new value.

Default run (62 steps): s = ' Apples, pears,,figs '. Evaluate ' | '.join(s.strip().lower().split(',')) one method call at a time, left to right. … ' | '.join(s.strip().lower().split(',')) → 'apples | pears | | figs'. And s is still ' Apples, pears,,figs '.

Simplified: Printable ASCII text up to 30 characters and up to four chained calls with short arguments. CPython searches with a faster algorithm (fastsearch) than the window-by-window scan shown, but the results are the same.

Educational simulation

Loading the simulation…

Why Raw Text Misleads

Real text files such as books, news articles, and social media posts contain punctuation and mixed capitalization. If text is split and counted without cleaning, the same word can appear in several forms. Its total count is then divided across multiple dictionary entries, and the apparent vocabulary becomes larger than it really is.

Key point

Text cleaning should happen before parsing when the goal is accurate word-frequency analysis.

What split() Actually Sees

The split() function looks for spaces and treats the text between spaces as individual tokens. It does not understand punctuation as separate from a word. When a punctuation mark is attached to a word, that punctuation remains part of the token. As a result, soft and soft! are different tokens and can become different dictionary keys.

How does the same sentence become separate tokens such as soft and soft!, and how can those tokens become different dictionary keys?

Worked example

Following Punctuation Through Counting

Suppose a text contains the tokens soft and soft! after splitting.

1
Observe the tokens
The tokens are not identical: one is soft and the other is soft! because the punctuation mark remains attached.
2
Create dictionary entries
A frequency dictionary can therefore store soft and soft! as separate keys.
3
Interpret the count
Looking only at the key soft would miss the occurrence stored under soft!.

Result

The word's frequency is fragmented across separate entries until punctuation is removed before parsing.

Why Case Splits Counts

Python treats uppercase and lowercase letters as distinct characters. A dictionary therefore treats Who and who as different keys. The same problem applies to other capitalization forms, such as Soft, soft, and SOFT. Without a consistent case-normalization step, occurrences of one word are divided across multiple entries.

How can Soft, soft, and SOFT remain separate entries before a consistent case-normalization step?

The methods lower() and upper() belong in the case-normalization stage. Choose one consistent case form before parsing and counting. The important principle is consistency: words that differ only in capitalization should be placed into one shared form rather than being counted under separate keys.

The Cleaning Pipeline

What happens to raw text as it moves through punctuation normalization, case conversion, splitting, and word counting?

1
Remove or normalize punctuation before parsing so attached marks do not remain part of tokens.
2
Choose a consistent case form with lower() or upper() before counting.
3
Use split() after cleaning so spaces separate the intended words.
4
Build the frequency dictionary from the cleaned tokens.

Best practice

Treat punctuation normalization and case normalization as preparation steps, not optional refinements. If they are postponed until after counting, the dictionary has already been fragmented and the original total for a word may be hidden across several keys.

Reading the Consequences

How does failing to normalize punctuation and capitalization divide one word's count across multiple dictionary entries?

Worked example

A Frequency Hidden Across Keys

Consider text in which It appears at the start of sentences, it appears elsewhere, and sun. appears with a period while sun appears without one.

1
Inspect capitalization
It and it are different dictionary keys because uppercase and lowercase letters are treated as distinct.
2
Inspect punctuation
sun. and sun are different tokens because split() preserves the period attached to sun.
3
Interpret the frequency data
The count for each word is divided across its forms, so the true frequency is not visible by checking only one key.
4
Clean before counting
Remove punctuation and choose a consistent case form before parsing so matching word forms can share one dictionary entry.

Result

Uncleaned text can hide true word frequencies, inflate apparent vocabulary size, and make it difficult to identify the most common words.

Text condition	What split() preserves	Effect on counting
Punctuation attached to a word	The punctuation remains part of the token	Forms such as soft and soft! become separate keys
Mixed capitalization	Uppercase and lowercase letters remain distinct	Forms such as Who and who become separate keys
Punctuation and case normalized first	Tokens use consistent forms	Matching word occurrences can be combined

Cleaning changes whether matching word occurrences are counted together or divided across dictionary entries.

Mistakes to Avoid

Common mistake
Assuming split() removes punctuation.
What it looks like
Expecting soft! to become soft automatically.
Why it fails
split() looks for spaces and treats the text between spaces as a token. It does not understand punctuation separately.
Do this instead
Normalize or remove punctuation before using the tokens for frequency analysis.
Common mistake
Treating capitalization as cosmetic during dictionary counting.
What it looks like
Counting Who and who as the same key without case normalization.
Why it fails
Python treats uppercase and lowercase letters as distinct characters, so the dictionary stores separate keys.
Do this instead
Choose a consistent case-normalization step with lower() or upper() before parsing.
Common mistake
Reading one dictionary key as the complete frequency of a word.
What it looks like
Checking soft while ignoring soft! and other variants.
Why it fails
Uncleaned text can split one word's occurrences across several entries.
Do this instead
Clean punctuation and case before building the frequency dictionary.
Common mistake
Skipping cleaning because the text looks understandable to a person.
What it looks like
Assuming a sentence is ready to count just because its meaning is clear.
Why it fails
A human can recognize related forms, but the dictionary treats distinct token strings as distinct keys.
Do this instead
Prepare the text systematically before parsing and counting.

Check Your Understanding

MEDIUM

A text produces the tokens Soft, soft, and soft!. Explain why these may become three dictionary entries, identify the two different cleaning problems involved, and describe the preparation steps needed before counting.

Hints

Compare the uppercase S with the lowercase s.
Look at the punctuation attached to soft!.
Place punctuation normalization and case normalization before split() and word counting.

Your attempt (not saved)

What do you think happens?

Before cleaning, will a frequency dictionary necessarily combine the occurrences represented by Who, who, and who! into one entry?

Yes, because they contain the same letters
No, because capitalization and punctuation can make them distinct keys

Reveal answer

Answer:

No, because capitalization and punctuation can make them distinct keys.

Uppercase and lowercase letters are treated as distinct, and punctuation attached to a word remains part of the token produced by split().

Key Takeaways

1
split() separates text at spaces and preserves punctuation attached to words.
2
Punctuation can make soft and soft! different tokens and different dictionary keys.
3
Python treats uppercase and lowercase letters as distinct, so Who and who can receive separate counts.
4
Uncleaned text fragments word frequencies and artificially inflates vocabulary size.
5
Remove or normalize punctuation and apply a consistent lower() or upper() case step before parsing and counting.

At a glance

split() preserves punctuation attached to words, so punctuation can create separate tokens.
Capitalization differences create separate dictionary keys because uppercase and lowercase letters are distinct.
Without cleaning, word-frequency counts become fragmented and vocabulary size appears artificially larger.
Use punctuation normalization and a consistent lower() or upper() case step before splitting and counting.

Related concepts

Text Parsing and Tokenization

More concepts starting with S