Concepts / Using re.search() and re.findall()

Using re.search() and re.findall()

Parentheses in regular expressions mark the portion of a match that findall() should extract and return.

  • Programming
Interactive lab

Try it: Regular expressions: a backtracking matcher

How Python's re module matches a pattern: it tries each start position, matches tokens one by one, lets greedy quantifiers take as much as possible and give it back when the rest fails (lazy ones take as little as possible), tries alternatives in order and records groups.

How it works

  1. search tries start positions 0, 1, 2 …; match tries only position 0; findall collects every non-overlapping match.
  2. Tokens are matched left to right: literals, ., classes like [a-c] and \d, and zero-width anchors ^, \$, \b.
  3. A greedy quantifier (* + ? {m,n}) takes as many repetitions as it can, then backtracks one at a time; a lazy one (*? +? ??) takes as few as possible and grows only when needed.
  4. Alternatives in a | b are tried in order; the first that lets the whole pattern match wins.
  5. Parentheses capture the text their part matched; captures are undone when the matcher backtracks past them.

Default run (63 steps): re.search('(\\w+)@(\\w+)\\.com', 'mail bo@ikb.com now'): try each start position until one matches. … re.search('(\\w+)@(\\w+)\\.com', 'mail bo@ikb.com now') → <re.Match object; span=(5, 15), match='bo@ikb.com'>

Simplified: A subset of re: no flags, lookarounds, backreferences, named groups or possessive quantifiers; pattern and text up to 24 printable ASCII characters. The trace stops after 1900 attempts ("too many steps") where real Python would keep backtracking.

Educational simulation

Loading the simulation…

One Match, Two Possible Results

A regular expression can match a larger piece of text while you keep only a smaller piece from inside it. The key tool for controlling what findall() returns is the capturing group: the part of the pattern enclosed in parentheses. Parentheses do not change the text that the pattern matches. They change which part of that match findall() returns.

X-:42pattern without parenthesesX-:42returned textX-:(42)pattern with a capturinggroup42captured textfindall()findall()
Which text does findall() return when the same matching pattern is written with and without a capturing group?

Tracing a Capturing Group

What do you think happens?

Suppose findall() uses the pattern X-:(\d+) on the text X-:42. What should the result contain: the full text X-:42 or only the number 42?

  • The full text X-:42

  • Only the text 42

Reveal answer
Answer:

Only the text 42

The complete pattern must match X-:42, but the parentheses identify 42 as the portion to extract. With one capturing group, findall() returns a list of strings containing the captured text.

Code walkthrough
import re

text = "X-:42"
values = re.findall(r"X-:(\d+)", text)
print(values)
  1. 1
    The pattern must match the prefix X-: followed by a number.
  2. 2
    The number is inside parentheses, so it is the capturing group.
  3. 3
    The result is a list containing the captured string.
  • The text outside the parentheses describes the surrounding text that must match.
  • The parentheses capture the number portion.
  • findall() returns the captured portion rather than the entire matched substring.
Output
['42']

Matching Versus Extracting

Think of the regular expression as having two jobs. First, the whole pattern decides whether the relevant text matches. Everything outside the parentheses still participates in that decision. Second, the parentheses identify the portion to keep when findall() builds its result. Therefore, a pattern can match a long string but return only a short substring from inside it.

X-:42complete matching text["42", "17", "8"]captured stringsX-:17complete matching textX-:8complete matching textcapture 42capture 17capture 8
How does findall() transform several complete matches into a list containing only the text inside the capturing group?

When several pieces of text match, findall() collects the extracted results. With one capturing group, the result is a list of strings. Without parentheses, the returned strings are the entire matching substrings instead. The matching process can therefore be the same while the returned result has different content.

Choosing search() or findall()

re.search()match objectre.findall()list of resultsfirst matching locationevery matching result
Does the operation give you the first matching location and match object, or a collection of extracted results from every match?
OperationResult focusUseful distinction
re.search()The first matching location and a match objectUse this when the question concerns whether and where a first match occurs.
re.findall()A collection of extracted resultsUse this when the question concerns all matching portions and the text captured by parentheses.

The two operations differ in the kind of result they provide.

The capturing-group rule in this article applies specifically to what findall() returns. re.search() is described differently: it gives the first matching location and a match object. Do not assume that choosing parentheses automatically makes both functions return the same kind of result.

Working with Extracted Values

Worked example

Extracting Two Parts from Each Line

A pattern must identify lines beginning with X-, followed by a colon, a label, and a number. Capture the label and number.

  1. 1
    Build the pattern

    Use two sets of parentheses so the label and number are separate captured portions. The surrounding characters still have to match.

  2. 2
    Apply findall()

    Because the pattern has more than one capturing group, findall() returns a list of tuples. Each tuple contains the captured values in the order of the groups.

  3. 3
    Interpret the values

    The captured number is still a string. Convert it to int when arithmetic or numeric comparison is needed.

Result

A list of tuples, with each tuple containing the captured label and number in pattern order.

python
Output
[('score', '42'), ('level', '7')]
score 43
level 8

The first result is a tuple because the pattern has two capturing groups. The tuple values are strings, including the value that looks like a number. Calling int() creates an integer for the later addition. Similarly, extracted decimal values need conversion to float when floating-point operations are required.

Mistakes Beginners Make

  • Common mistake

    Expecting findall() to return the whole match when a capturing group is present.

    What it looks like

    The pattern X-:(\d+) matches X-:42, but findall() returns 42 inside its result list.

    Why it fails

    Parentheses tell findall() to return only the enclosed portion.

    Do this instead

    Separate the idea of matching the complete pattern from extracting the captured portion.

  • Common mistake

    Assuming captured numbers are already numeric.

    What it looks like

    The extracted value is '42', not the integer 42.

    Why it fails

    findall() returns extracted values as strings.

    Do this instead

    Convert the value explicitly with int() or float() when the later operation requires that type.

  • Common mistake

    Treating multiple capturing groups like one capturing group.

    What it looks like

    A pattern with a label group and a number group returns tuples rather than a simple list of strings.

    Why it fails

    findall() represents multiple captured values from each match as a tuple.

    Do this instead

    Unpack each tuple in the same order as the capturing groups appear in the pattern.

  • Common mistake

    Indexing the result without checking whether anything matched.

    What it looks like

    A line that does not match produces an empty list.

    Why it fails

    There is no element to access when findall() returns no matches.

    Do this instead

    Verify that the returned list is non-empty before accessing one of its elements.

Practice: Predict the Result

EASY

Consider the pattern X-:(\d+) applied to the text X-:12 X-:30. Predict the result of findall() before checking the answer. Then state what conversion is needed if you want to add 5 to each extracted value.

Hints
  • The number is inside the only capturing group.
  • One capturing group produces a list of strings.
  • Convert each string to int before performing integer arithmetic.
Worked example

Practice Check

Determine the result shape for X-:(\d+) applied to X-:12 X-:30, and identify the conversion needed for addition.

  1. 1
    Identify the group

    The digits are inside one pair of parentheses, so only the digit text is captured.

  2. 2
    Identify the result shape

    One capturing group produces a list of strings.

  3. 3
    Prepare for arithmetic

    Convert each extracted string to int before adding 5.

Result

The result is ['12', '30']; convert the strings with int() before arithmetic.

Key Takeaways

  1. 1
    The complete regular expression must match, but parentheses identify the part findall() should return.
  2. 2
    With one capturing group, findall() returns a list of strings.
  3. 3
    With multiple capturing groups, findall() returns a list of tuples in group order.
  4. 4
    Extracted values are strings, so convert them to int or float when numeric work is needed.
  5. 5
    Check that findall() returned a non-empty list before accessing an element.
  6. 6
    re.search() focuses on the first matching location and a match object, while re.findall() provides a collection of extracted results.

At a glance

  • Parentheses create capturing groups that control what findall() extracts.
  • The surrounding pattern still determines whether the complete match succeeds.
  • One group produces strings; multiple groups produce tuples.
  • Convert extracted strings before numeric operations, and check for an empty result first.
  • Use the distinction between re.search() and re.findall() to choose between a first match object and a collection of extracted results.

Related Topics

Related concepts

More concepts starting with U