Fetching Web Content with urllib

Fetching web content for URL extraction involves two connected tasks.

Programming

From HTML to URLs

Fetching web content for URL extraction involves two connected tasks. First, a program obtains HTML content with urllib. Next, a regular expression searches that HTML and isolates the URL text. The important mechanism in the extraction step is the capturing group: parentheses identify the part of a larger match that should be returned.

Key point

A regular expression can match a larger HTML fragment while a capturing group selects only the substring you want to extract.

Capturing the URL Text

Consider the generated pattern <a href="(.*?)">. The pattern describes an anchor-like HTML fragment. The parentheses around .*? create a capturing group. The complete match includes the surrounding HTML text, but the captured substring is the text between the opening quote after href= and the next closing quote. The question mark makes the repetition non-greedy, so .*? stops at the first closing quote rather than consuming too much text.

Which part of the HTML pattern is the full match, and which part is isolated by the parentheses?

python

pattern = r'<a href="(.*?)">'

pattern = r'<a href="(.*?)">'

Tracing findall Results

Worked example

Extracting two links

Apply the pattern <a href="(.*?)"> to the generated HTML fragment <a href="/home">Home</a><a href="/about">About</a>.

1
Locate the first match
The pattern matches the first anchor-like fragment and the capturing group contains /home.
2
Continue through the HTML
The search continues after the first matching fragment and finds the second one. Its capturing group contains /about.
3
Return the captured values
Because the pattern has a capturing group, re.findall() returns the captured URL text rather than the complete matching HTML fragments.

Result

The extracted values are /home and /about.

python

import re

html = '<a href="/home">Home</a><a href="/about">About</a>'
pattern = r'<a href="(.*?)">'
urls = re.findall(pattern, html)
print(urls)

import re

html = '<a href="/home">Home</a><a href="/about">About</a>'
pattern = r'<a href="(.*?)">'
urls = re.findall(pattern, html)
print(urls)

Output

['/home', '/about']

How does re.findall() move through the HTML and return the URL captured by the group instead of the entire matching tag?

Key point

re.findall() is useful here because its result consists of the captured groups. The surrounding anchor text is used to identify each match but is not returned as the extracted URL when the group is present.

Request and Certificate Context

The extraction pattern operates on HTML content obtained by the web request. When the target site has an invalid certificate, certificate verification must be disabled in the fetching code by using an SSL context. The context belongs to the request stage; once HTML has been obtained, the regular expression processes that HTML separately.

python

import ssl

context = ssl._create_unverified_context()
# Use context when making the urllib request to a site with an invalid certificate.
# Then apply the regular-expression pattern to the returned HTML.

import ssl

context = ssl._create_unverified_context()
# Use context when making the urllib request to a site with an invalid certificate.
# Then apply the regular-expression pattern to the returned HTML.

Debugging a Failed Match

When URL extraction fails, compare the pattern with the actual HTML rather than assuming that the pattern describes the input correctly. Trace where the pattern begins matching, what text the group captures, and where the non-greedy expression stops. If the actual HTML differs from the expected anchor-like structure, the pattern may stop matching before a URL is captured or may produce a different captured substring.

Where does matching stop when the actual HTML differs from the expected pattern, and what text is captured at each successful match?

Common mistake
Leaving out the capturing parentheses.
What it looks like
Using a pattern that describes the link but does not place parentheses around the URL portion.
Why it fails
The extraction no longer identifies the URL substring as a captured group for re.findall() to return.
Do this instead
Place parentheses around the part of the match that should become the extracted URL.
Common mistake
Using a greedy wildcard instead of .*?.
What it looks like
Using .* between the opening and closing quotes.
Why it fails
The expression can match too much instead of stopping at the first closing quote.
Do this instead
Use the non-greedy quantifier .*? for this URL-extraction pattern.
Common mistake
Debugging only the regular expression and not the input HTML.
What it looks like
Assuming every link begins exactly with <a href=".
Why it fails
A pattern can fail when the actual HTML differs from the structure it expects.
Do this instead
Inspect the actual HTML and trace the first point where it differs from the pattern.
Common mistake
Treating certificate problems as regex problems.
What it looks like
Changing the capturing group when the web request cannot obtain the HTML because of an invalid certificate.
Why it fails
Certificate verification occurs during fetching, before regex processing.
Do this instead
Configure an SSL context for the request when fetching from a site with an invalid certificate.

A Reliable Debugging Routine

1
Confirm that the web request produced HTML content. If the site has an invalid certificate, review the SSL context used for the request.
2
Write down the exact HTML fragment that should contain a URL.
3
Mark the opening and closing boundaries that the pattern expects.
4
Check that parentheses surround the substring to be returned.
5
Check that .*? is used when the match must stop at the first closing quote.
6
Run re.findall() and inspect the returned captured values.
7
If the result is empty or unexpected, compare the first failed HTML fragment with the pattern character by character.

EASY

For the generated HTML fragment <a href="/docs">Docs</a><a href="/contact">Contact</a>, identify the two strings that should be returned by re.findall() when the pattern is <a href="(.*?)">. Then explain why the complete anchor text is not returned.

Hints

Find the text between each pair of quotes after href=.
The parentheses identify the captured part of each match.
The surrounding <a href=" and "> text helps define the match but is not the captured URL.

Your attempt (not saved)

Key Takeaways

1
Parentheses create capturing groups that isolate a substring from a larger regular-expression match.
2
The non-greedy quantifier .*? helps URL extraction stop at the first closing quote.
3
re.findall() returns the captured groups, making it suitable for collecting URLs from matching HTML fragments.
4
Regex debugging requires comparing the expected pattern with the actual HTML and tracing each match.
5
An SSL context is used at the fetching stage when certificate verification must be disabled for a site with an invalid certificate.

At a glance

Capturing parentheses determine which substring is extracted from a larger HTML match.
Use .*? so URL matching stops at the first closing quote rather than matching too much.
re.findall() returns the captured URL values instead of the entire matching tags.
Compare the pattern with actual HTML when extraction fails.
Use an SSL context during fetching when a target site has an invalid certificate.

More concepts starting with F

From HTML to URLs

Capturing the URL Text

Tracing findall Results

Request and Certificate Context

Debugging a Failed Match

A Reliable Debugging Routine

Key Takeaways

At a glance

Related Topics