Concepts / Using findall() and search() for Pattern Matching

Using findall() and search() for Pattern Matching

Regular expressions are search strings with special characters that communicate matching criteria and extraction rules to a regex system.

  • Programming
Interactive lab

Try it: Regular expressions: a backtracking matcher

How Python's re module matches a pattern: it tries each start position, matches tokens one by one, lets greedy quantifiers take as much as possible and give it back when the rest fails (lazy ones take as little as possible), tries alternatives in order and records groups.

How it works

  1. search tries start positions 0, 1, 2 …; match tries only position 0; findall collects every non-overlapping match.
  2. Tokens are matched left to right: literals, ., classes like [a-c] and \d, and zero-width anchors ^, \$, \b.
  3. A greedy quantifier (* + ? {m,n}) takes as many repetitions as it can, then backtracks one at a time; a lazy one (*? +? ??) takes as few as possible and grows only when needed.
  4. Alternatives in a | b are tried in order; the first that lets the whole pattern match wins.
  5. Parentheses capture the text their part matched; captures are undone when the matcher backtracks past them.

Default run (63 steps): re.search('(\\w+)@(\\w+)\\.com', 'mail bo@ikb.com now'): try each start position until one matches. … re.search('(\\w+)@(\\w+)\\.com', 'mail bo@ikb.com now') → <re.Match object; span=(5, 15), match='bo@ikb.com'>

Simplified: A subset of re: no flags, lookarounds, backreferences, named groups or possessive quantifiers; pattern and text up to 24 printable ASCII characters. The trace stops after 1900 attempts ("too many steps") where real Python would keep backtracking.

Educational simulation

Loading the simulation…

A Pattern Can Ask Two Different Questions

Regular expressions are search strings containing special characters. Those characters communicate matching criteria and extraction rules to a regex system. When the same pattern is applied to text, search() answers whether and where one match can be found, while findall() is used to collect multiple matches. The choice depends on whether the task is to locate one occurrence or gather the occurrences that satisfy the pattern.

Textcat bat catsearch()one matching occurrenceOne matchcatPatterncatfindall()all matching occurrencesMultiple matchescat, catapplyuseapplyuselocatescollects
Given the same text and pattern, how does one-match searching differ from collecting multiple matches?
python

In this example, both operations use the same text and pattern. The search() operation is the focused question: can the pattern locate a match? The findall() operation is the collection question: which occurrences in the text match? The regular expression itself determines what counts as a match; the operation determines how matching results are used.

Read a Pattern as a Set of Constraints

A regex pattern can constrain position, character identity, repetition, or extraction. Anchors such as ^ and $ refer to positions at the start or end of a line. A wildcard such as . represents any character. Character sets restrict a position to one character from a defined set or range. Quantifiers control how many times a preceding pattern repeats. Parentheses identify capture groups so that selected parts of a larger match can be extracted.

Pattern featureQuestion it adds
^ or $Must the match be at the start or end of a line?
.Can any character occupy this position?
* , + , ?How many times may the preceding pattern repeat?
*? , +? , ??Can the repetition be non-greedy?
[...]Which individual characters are allowed?
(...)Which part should be captured for extraction?
\d, \s, \bIs the pattern concerned with digits, whitespace, or a word boundary?

Regex features describe different constraints on a possible match.

Control the Scan with Repetition

Quantifiers control repetition. The symbols *, +, and ? express repetition choices, while *?, +?, and ?? are their non-greedy forms. A greedy quantifier matches as many characters as possible. A non-greedy quantifier, indicated by the trailing ?, matches as few characters as possible. This difference matters when more than one span of text could satisfy the surrounding pattern.

Start scantext positionGreedyextend as far as possibleLonger spanpossible matchNon-greedyextend as little aspossibleShorter spanpossible matchchoosechooseas many as possibleas few as possible
As a regex scans text, how does a greedy quantifier differ from a non-greedy quantifier?
Worked example

Choosing a repetition style

A pattern must match a repeated portion of text, but the surrounding pattern may permit either a longer or shorter span.

  1. 1
    Use a greedy quantifier

    The regex gives the repeated portion permission to consume as many characters as possible.

  2. 2
    Use a non-greedy quantifier

    Add the trailing question mark form so the repeated portion attempts to consume as few characters as possible.

  3. 3
    Compare the spans

    The greedy and non-greedy forms can select different spans even though they describe the same general repeated pattern.

Result

Choose greedy repetition when the broadest possible span is wanted and non-greedy repetition when the narrowest possible span is wanted.

Restrict Characters with Sets

A character set matches a single character from the set described inside square brackets. For example, [aeiou] limits a position to the listed vowels. A range such as [a-z0-9] describes lowercase letters from a through z and digits from 0 through 9. A negated set such as [^A-Za-z] describes characters outside the listed uppercase and lowercase letter ranges. The set changes which characters are eligible at each position; it does not by itself describe a whole multi-character word.

[aeiou]a, e, i, o, uVowelsaccepted characters[a-z0-9]lowercase letters or digitsLetters and digitsaccepted characters[^A-Za-z]not an uppercase orlowercase letterNonlettersaccepted charactersacceptsacceptsaccepts
Which characters are accepted by a vowel set, an alphanumeric range, and a negated letter set?
python

These three patterns ask different character questions. The vowel set selects listed vowels. The range accepts a lowercase letter or a digit. The negated letter set accepts a character that is not in either letter range. The caret has a different role inside a set than it has at the beginning of a pattern: in the set example it expresses negation, while the anchor form refers to a start position.

Mark Positions with Anchors and Boundaries

Anchors match positions rather than ordinary characters. The ^ anchor restricts a match to the start of a line, and the $ anchor restricts a match to the end of a line. A word boundary is represented by \b. It identifies a boundary position associated with a word, while its negation describes a position that is not a word boundary. These tools change where a match is allowed to begin or end.

^start positionLine starteligible position\bword boundaryWord boundaryeligible position$end positionLine endeligible positionrequiresrequiresrequires
How do ^, $, and a word boundary change which positions in text are eligible for a match?
Worked example

Adding a position requirement

Match a word only when it occurs at the beginning of a line rather than merely appearing somewhere in the line.

  1. 1
    Start with the word pattern

    The word pattern describes the characters that should be matched.

  2. 2
    Add ^

    Place the start anchor before the word pattern to require the match to begin at the start of the line.

  3. 3
    Use $ when the end matters

    Place the end anchor after the pattern when the match must finish at the end of the line.

  4. 4
    Use \b for a word boundary

    Use a word boundary when the important condition is the transition at a word edge rather than the absolute start or end of the line.

Result

Anchors and boundaries do not add ordinary characters to the match; they restrict the positions where the match may occur.

Capture the Useful Substring

Parentheses create capture groups. A pattern can therefore match a larger string while identifying particular subsets inside it for extraction. The full match represents the larger pattern, and each parenthesized part represents a captured subgroup within that match.

Order:surrounding textFull matchentire matched span(digits)capture this subsetCapture groupparenthesized subsetSuffixsurrounding textpart ofdefinespart ofwithin
When a pattern matches a larger string, which part is the full match and which part is stored as a capture group?
python

In the pattern item-(\d+), the complete pattern describes the larger item-482 match, while the parentheses identify the digit portion as a capture group. Parentheses are therefore useful when matching context is necessary but only a subset of that context needs to be extracted.

Choose the Right Character Class

Shorthand sequences provide convenient ways to describe common matching concerns. The sequence \d represents digits, \s represents whitespace, and \b represents word boundaries. Their negations describe the corresponding opposite conditions. These classes answer different questions: whether a position contains a digit, whether it contains whitespace, or whether it is a boundary between word-related regions.

\swhitespaceWhitespacecharacter condition\ddigitDigitcharacter condition\bword boundaryBoundaryposition condition\Snot whitespace\Dnot a digit\Bnot a word boundarymatchesmatchesmatchesnegatesnegatesnegates
How do whitespace, digits, word boundaries, and their negations differ when applied to text?
PatternFocusNegated form
\dDigits\D: not digits
\sWhitespace\S: not whitespace
\bWord boundary position\B: not a word boundary

Shorthand classes distinguish character conditions from boundary-position conditions.

Mistakes Beginners Make

  • Common mistake

    Treating ^ and $ as ordinary characters to be found in the text.

    What it looks like

    Using ^word when the intention is to search for a literal caret followed by the letters word.

    Why it fails

    Anchors describe positions at the start or end of a line rather than ordinary character content.

    Do this instead

    Use anchors when the location of the match matters, and use a character pattern when the character itself matters.

  • Common mistake

    Assuming a character set matches a complete word.

    What it looks like
    [aeiou]
    Why it fails

    A character set describes allowed characters for a position; it does not automatically describe a multi-character word.

    Do this instead

    Combine the set with an appropriate repetition quantifier when repeated matching is required.

  • Common mistake

    Ignoring greediness when a repeated pattern can span different lengths.

    What it looks like

    Using * when the shortest possible repeated span is required.

    Why it fails

    Greedy quantifiers match as many characters as possible.

    Do this instead

    Use the corresponding non-greedy form with a trailing question mark when the fewest possible characters should be matched.

  • Common mistake

    Confusing the full match with a capture group.

    What it looks like
    Pattern item-(\d+)
    Why it fails

    The complete pattern can match a larger string, while the parentheses identify only the selected subset.

    Do this instead

    Use the full match for the complete span and the capture group for the parenthesized subset.

  • Common mistake

    Using a digit class when the requirement is a word boundary.

    What it looks like
    Replacing \b with \d.
    Why it fails

    The digit shorthand concerns digit characters, whereas a word boundary concerns a position.

    Do this instead

    Select the shorthand class that describes the actual condition: \d for digits, \s for whitespace, and \b for word boundaries.

Practice the Decision

MEDIUM

You need to inspect the text "ID 482 ID 917". Decide which operation and pattern you would use for each task: locate one occurrence of ID followed by digits; collect every digit sequence; capture only the digits from each ID; and require the pattern to occur at the start of the line.

Hints
  • Use search() when the task asks you to locate one match.
  • Use findall() when the task asks you to collect multiple matches.
  • Use parentheses around the part that should be captured.
  • Use \d for digits and ^ for the start position.
Worked example

A pattern-building route

Construct a pattern for an ID that begins with ID, is followed by whitespace, and then contains digits. Capture the digits rather than treating them as incidental context.

  1. 1
    Match the literal prefix

    Begin with the characters ID because they identify the surrounding context.

  2. 2
    Describe whitespace

    Add \s to require a whitespace position after the prefix.

  3. 3
    Describe digits

    Add \d with a repetition quantifier to describe the digit portion.

  4. 4
    Capture the useful portion

    Place parentheses around the repeated digit pattern so the digits form a capture group.

  5. 5
    Choose the operation

    Use search() for one located ID or findall() when all matching IDs are needed.

Result

The pattern structure is ID followed by whitespace and a parenthesized repeated digit pattern; the operation depends on whether one match or multiple matches are required.

A Reliable Matching Workflow

  1. 1
    State whether the task needs one located match or a collection of matches.
  2. 2
    Mark positional requirements with ^, $, or a word boundary.
  3. 3
    Choose a character set or shorthand class for the characters involved.
  4. 4
    Add a quantifier to control repetition, then decide whether greedy or non-greedy behavior is appropriate.
  5. 5
    Add parentheses around any subset that must be extracted.
  6. 6
    Test the complete pattern against the intended text and check whether the selected span and captured subset are different in the expected way.

A useful mental model is to build the pattern from the outside in. First decide where a match may occur, then specify which characters may appear, then control repetition, and finally mark the subset that should be captured. After that, choose search() or findall() according to the number of matching occurrences the task requires.

Key Takeaways

  • search() is suited to locating one matching occurrence, while findall() is suited to collecting multiple matching occurrences.
  • Anchors and word boundaries constrain positions; character sets and shorthand classes constrain character conditions.
  • Greedy quantifiers match as many characters as possible, while non-greedy forms match as few as possible.
  • Parentheses create capture groups that identify useful subsets inside a larger full match.
  • Build patterns by deciding the position, allowed characters, repetition, and extraction requirements.

Related Topics

Related concepts

More concepts starting with U