Concepts / Parsing and Processing Text Data

Parsing and Processing Text Data

Text from a file, user input, or another data source often arrives as one long string.

  • Programming

From Text to Usable Pieces

Text from a file, user input, or another data source often arrives as one long string. That form is useful for receiving the data, but tasks such as counting words, searching for a particular word, rearranging words, or analyzing separate values require smaller pieces. Python's split method converts a string into a new list of words or other delimiter-separated values.

Key point

Use split when you need to work with words or other logical units. Use list when you need to work with individual characters.

Whitespace Boundaries

Call split on a string with no argument to break the text at whitespace. In this context, whitespace includes spaces, tabs, and newlines. The result is a new list in which each word is a separate string. The whitespace that served as the boundary is not included in the resulting list.

sentencered blue greenredlist itembluelist itemgreenlist itemsplit on whitespacesplit on whitespacesplit on whitespace
What changes when a whitespace-separated string is transformed into a list of words?
Worked example

Separating a Sentence

Convert the string red blue green into a list using split with no argument.

  1. 1
    Start with one string

    The text is a single string containing three words separated by whitespace.

  2. 2
    Call split

    With no argument, split uses whitespace as the boundary between words.

  3. 3
    Inspect the result

    The returned list contains three separate strings: red, blue, and green. The spaces are not list elements.

Result

A new list containing the three words red, blue, and green

Positions in the Result

After splitting, each word occupies one position in the resulting list. Square-bracket index notation retrieves the item at a chosen position. Python starts counting positions at 0, so the first word is at index 0, the second is at index 1, and the third is at index 2.

0red1blue2green
How does each word map to its list position, and which word does a given index access?

What do you think happens?

Suppose a split list contains red at index 0, blue at index 1, and green at index 2. Which word does index 1 access?

  • red

  • blue

  • green

Reveal answer
Answer:

blue

Index counting starts at 0, so index 1 identifies the second item in the list.

Indexing does not change the list. It selects the item stored at one position. The same square-bracket notation works with a list created by split and with other lists.

Custom Delimiters

Some text does not use spaces between its values. Values may be separated by commas, hyphens, colons, or other characters. Pass the desired delimiter as an argument to split. Python breaks the string wherever that delimiter appears, and the delimiter itself is removed from the resulting list.

red-blue-greenredbluegreensplit on hyphensplit on hyphensplit on hyphen
How does selecting a delimiter change the boundaries of the pieces produced from a string?
Worked example

Parsing Hyphen-Separated Values

Convert the string red-blue-green into separate values by using the hyphen as the delimiter.

  1. 1
    Identify the boundary

    The hyphen separates the three values, so it is the character that should be supplied to split.

  2. 2
    Split at each hyphen

    Python scans the string and divides it wherever a hyphen occurs.

  3. 3
    Remove the delimiter from the result

    The hyphens identify the boundaries but do not appear as list elements.

Result

A new list containing red, blue, and green

In the real world

A comma-separated text format can be processed by choosing the comma as the delimiter. The same pattern applies to other structured text when its fields are separated by a known character.

Characters Versus Words

The list function and the split method both produce lists, but they divide a string at different levels. list breaks the string into individual characters. Spaces and punctuation are included as character elements. split breaks the string into words when used without an argument, or into delimiter-separated units when given a delimiter. The boundaries are removed from the split result.

red bluered bluercharacterredwordspacecharacterbluewordlistlistsplitsplit
Why does list produce characters while split produces larger word or delimiter-separated units?
ToolBoundaryResult
listEvery characterA list of individual characters, including spaces and punctuation
split with no argumentWhitespaceA list of words with whitespace removed as boundaries
split with a delimiterThe supplied delimiterA list of delimiter-separated units with the delimiter removed

Choose the operation according to the level of detail you need.

Mistakes to Avoid

  • Common mistake

    Expecting split with no argument to separate characters

    What it looks like

    Treating a sentence processed by split as if each letter were a separate list item

    Why it fails

    The no-argument form uses whitespace as the boundary and produces words, not individual characters.

    Do this instead

    Use list when the task requires individual characters.

  • Common mistake

    Expecting list to produce words

    What it looks like

    Using list on a sentence and looking for one item per word

    Why it fails

    list breaks the string into individual characters, and spaces also appear as character elements.

    Do this instead

    Use split with no argument for whitespace-separated words.

  • Common mistake

    Using the wrong delimiter

    What it looks like

    Trying to split red-blue-green by supplying a comma as the delimiter

    Why it fails

    Python breaks the string at the character supplied to split. A delimiter that is not present does not identify the hyphen boundaries.

    Do this instead

    Pass the character that actually separates the values.

  • Common mistake

    Starting the first index at 1

    What it looks like

    Treating index 1 as the first word

    Why it fails

    Python positions start at 0.

    Do this instead

    Use index 0 for the first item, index 1 for the second item, and continue from there.

  • Common mistake

    Assuming split changes the original string

    What it looks like

    Expecting the original sentence itself to become a list after calling split

    Why it fails

    split returns a new list and does not modify the original string.

    Do this instead

    Keep or assign the returned list when you need to work with the separated items.

Best practice

Before splitting, identify the unit your next operation needs. If the unit is a word, use split with no argument for whitespace-separated text. If the unit is a field separated by a known character, pass that character as the delimiter. If the unit is a single character, use list instead.

Practice the Transformation

EASY

A string contains the text north south east. Describe the list produced by splitting on whitespace, identify the index of south, and state which word is returned by index 2.

Hints
  • Write the words in their left-to-right order.
  • Begin counting positions at 0.
  • Index 2 refers to the third item.
MEDIUM

A string contains the text 2026:science:python. Which delimiter should you supply to split so that the result contains three separate values? What happens to the colons in the result?

Hints
  • Look for the character repeated between the values.
  • The delimiter marks boundaries but is not retained as a result item.

Key Takeaways

  • The split method returns a new list from a string.
  • With no argument, split uses whitespace such as spaces, tabs, and newlines as boundaries.
  • A delimiter argument lets split separate values at a chosen character, and the delimiter is removed from the result.
  • Use square-bracket index notation to access list items, remembering that the first position is index 0.
  • Use list for individual characters and split for words or other delimiter-separated units.

Related Topics

Related concepts

More concepts starting with P