Concepts / Regular Expression Fundamentals

Regular Expression Fundamentals

A regular expression can describe the surrounding HTML, while a capturing group identifies the part you actually want to extract.

  • Programming
Interactive lab

Try it: Regular expressions: a backtracking matcher

How Python's re module matches a pattern: it tries each start position, matches tokens one by one, lets greedy quantifiers take as much as possible and give it back when the rest fails (lazy ones take as little as possible), tries alternatives in order and records groups.

How it works

  1. search tries start positions 0, 1, 2 …; match tries only position 0; findall collects every non-overlapping match.
  2. Tokens are matched left to right: literals, ., classes like [a-c] and \d, and zero-width anchors ^, \$, \b.
  3. A greedy quantifier (* + ? {m,n}) takes as many repetitions as it can, then backtracks one at a time; a lazy one (*? +? ??) takes as few as possible and grows only when needed.
  4. Alternatives in a | b are tried in order; the first that lets the whole pattern match wins.
  5. Parentheses capture the text their part matched; captures are undone when the matcher backtracks past them.

Default run (63 steps): re.search('(\\w+)@(\\w+)\\.com', 'mail bo@ikb.com now'): try each start position until one matches. … re.search('(\\w+)@(\\w+)\\.com', 'mail bo@ikb.com now') → <re.Match object; span=(5, 15), match='bo@ikb.com'>

Simplified: A subset of re: no flags, lookarounds, backreferences, named groups or possessive quantifiers; pattern and text up to 24 printable ASCII characters. The trace stops after 1900 attempts ("too many steps") where real Python would keep backtracking.

Educational simulation

Loading the simulation…

From HTML to URLs

Suppose an HTML document contains several links and your goal is to collect only their URL values. A regular expression can describe the surrounding HTML, while a capturing group identifies the part you actually want to extract. The important mechanism is the relationship between the complete match and the parenthesized substring inside it.

What do you think happens?

A pattern contains a larger match with a parenthesized URL portion. What should re.findall() return when used with that pattern?

  • The complete HTML fragment for every match

  • Only the text captured by the parentheses

  • Only the first character of every match

Reveal answer
Answer:

Only the text captured by the parentheses

In this extraction pattern, re.findall() returns the captured groups rather than the entire pattern match.

Tracing a URL Pattern

href="opening HTML attribute(.*?)captured URL text"closing quotethenthen
What does each part of a URL-extraction pattern match, and where does the captured substring begin and end?

A regular expression can match more text than you want to return. Parentheses mark a capturing group, so the text matched inside those parentheses is isolated from the rest of the pattern. In a URL pattern, the surrounding text can identify an href attribute while the group identifies the URL value inside the quotation marks.

The characters .*? are especially important in this pattern. The dot and asterisk describe a span of text, while the question mark makes that span non-greedy. The non-greedy form stops at the first closing quote instead of continuing across later text and matching too much.

python

Capturing the Substring

href="https://example.test"complete pattern matchhttps://example.testcaptured groupURL listcaptured valuesparentheses isolatefindall returns
How does the parenthesized group isolate a URL from the complete HTML match, and where does that substring appear in the result?
Worked example

Separating the Match from the Result

Use a pattern with one capturing group to extract the URL from an HTML link.

  1. 1
    Describe the surrounding text

    The pattern begins by matching the href attribute marker and ends by matching its closing quote.

  2. 2
    Create the group

    The parentheses around .*? identify the substring between those quotes as the captured value.

  3. 3
    Control the span

    The non-greedy quantifier makes the group stop at the first closing quote rather than consuming too much text.

  4. 4
    Read the extraction

    When re.findall() processes the pattern, it returns the captured URL values instead of the complete href fragments.

Result

The surrounding match locates the attribute, and the capturing group supplies the URL value returned by re.findall().

Code walkthrough
import re

html = '<a href="https://example.test">Example</a>'
pattern = r'href="(.*?)"'
urls = re.findall(pattern, html)
print(urls)
  1. 1
    The HTML string provides the text to search.
  2. 2
    The pattern matches the href marker, captures the text inside the quotes, and then matches the closing quote.
  3. 3
    re.findall() returns the captured URL value in a list.
  4. 4
    The printed result contains the URL rather than the complete href attribute.
Output
["https://example.test"]

The findall Data Flow

HTML contentlinks and attributesURL patternone capturing groupre.findall()collects captured valuesURL listextracted substringssearchapplyreturn groups
How does HTML text flow through the regular expression into the list of extracted URLs?

The extraction process has three useful stages. First, HTML content is supplied as the text to search. Next, the regular expression locates the relevant attribute and captures the substring inside it. Finally, re.findall() collects the captured values into a list. Its behavior is especially useful here because it returns the captured groups, not the entire pattern match.

python
Output
["https://one.test", "https://two.test"]

Debugging Against Actual HTML

href='URL'actual attribute usessingle quoteshref="(.*?)"pattern expects doublequoteshref="URL"attribute matches patternformhref="(.*?)"same pattern now matchescomparecompare
Which part of the regex matches the actual HTML, and where does matching fail?

A failed extraction should be debugged by comparing the pattern with the actual HTML, not by staring at the pattern in isolation. Trace the match from left to right. Check the attribute marker, check the quote character, check the captured span, and check the closing quote. A mismatch in the surrounding HTML can prevent the capturing group from being reached at all.

python
Output
[]
Best practice

When the result is empty, inspect the exact HTML text first. Confirm that the pattern's surrounding text appears in the same form, then verify that the quoted portion is where the pattern expects it. If the pattern matches too much, inspect whether .*? was used; the non-greedy form is essential for stopping at the first closing quote.

SSL Contexts for Requests

URL extraction may begin with fetching HTML from an HTTPS site. If the site has an invalid certificate, certificate verification must be disabled in the fetching code by using an SSL context. This setting concerns the HTTPS request that obtains the HTML; it does not change what the regular expression captures after the HTML has been received.

  1. 1
    Use an SSL context when the request must handle a site with an invalid certificate.
  2. 2
    Obtain the HTML content through that configured request.
  3. 3
    Inspect the actual HTML returned by the request.
  4. 4
    Apply the regular expression and verify the captured URL values.

Common Extraction Mistakes

  • Common mistake

    Using a greedy span when extracting a quoted URL

    What it looks like

    A pattern uses .* instead of .*? between the opening and closing quotes.

    Why it fails

    The match can continue farther than the first closing quote and consume too much text.

    Do this instead

    Use the non-greedy quantifier .*? so the URL extraction stops at the first closing quote.

  • Common mistake

    Expecting re.findall() to return the complete HTML match

    What it looks like

    The pattern matches an href attribute and also contains a parenthesized URL group.

    Why it fails

    With a capturing group, re.findall() returns the captured group values rather than the entire pattern match.

    Do this instead

    Read the returned list as the extracted substrings isolated by the parentheses.

  • Common mistake

    Debugging only the regular expression

    What it looks like

    The pattern expects one form of quoted attribute, but the actual HTML uses another form.

    Why it fails

    The pattern can fail before the capturing group is reached when the surrounding HTML does not match.

    Do this instead

    Compare each part of the pattern with the actual HTML returned by the request.

  • Common mistake

    Treating an HTTPS certificate problem as a regex problem

    What it looks like

    The request does not obtain the intended HTML because the site's certificate is invalid.

    Why it fails

    There is no correct HTML text for the regex to inspect until the request is handled.

    Do this instead

    Configure an SSL context for the invalid-certificate case, then debug extraction against the HTML that was actually received.

Practice the Trace

EASY

Given the HTML text <a href="https://one.test">One</a> and the pattern href="(.*?)", identify the complete portion matched by the pattern, the substring captured by the parentheses, and the value that re.findall() places in its result.

Hints
  • Separate the surrounding href text from the text inside the parentheses.
  • The non-greedy group stops at the first closing quote.
  • re.findall() returns the captured URL value rather than the complete href fragment.
MEDIUM

A URL extraction returns an empty list. The request used an SSL context for a site with an invalid certificate, but the returned HTML uses single quotes around href values while the pattern expects double quotes. Trace the failure and state which part must be investigated first.

Hints
  • First separate the request stage from the matching stage.
  • Confirm that HTML was received before examining the pattern.
  • Compare the quote characters in the actual HTML with the quote characters expected by the pattern.

Key Takeaways

  1. 1
    Parentheses create capturing groups that isolate specific substrings from a larger regular expression match.
  2. 2
    The non-greedy quantifier .*? is essential when a URL must stop at the first closing quote.
  3. 3
    re.findall() returns captured group values, making it useful for collecting URLs from HTML.
  4. 4
    Reliable debugging means tracing the pattern against the actual HTML rather than inspecting the pattern alone.
  5. 5
    An SSL context can be used to disable certificate verification when fetching from a site with an invalid certificate.

At a glance

  • Capturing groups use parentheses to isolate the substring that matters.
  • Use .*? for quoted URL extraction so matching stops at the first closing quote.
  • re.findall() returns captured values instead of the entire pattern match.
  • Debug extraction by comparing every pattern component with the actual HTML.
  • Use an SSL context to handle certificate verification when fetching from sites with invalid certificates.

Related Topics

More concepts starting with R