Handling Errors in Web Scraping

BeautifulSoup's soup('a') method finds all anchor tags in parsed HTML; tag.get('href', None) safely extracts the href attribute from each tag.

Programming

Interactive lab

Try it: Parsing HTML with BeautifulSoup

How BeautifulSoup(html, 'html.parser') turns raw HTML text into a tree of Tag objects, tolerating unclosed and mis-nested tags, and how soup('a'), find() and find_all() with class_ / attrs filters pick tags out of it so .get('href', None), tag['href'], .attrs, .contents, .get_text() and .string can extract their data.

How it works

html.parser reads the text left to right and reports start tags (with their attributes), end tags, text, comments and declarations.
Each start tag becomes a Tag inside the tag that is currently open; void elements such as <br> and <img> are closed at once; class and rel values are split into lists.
An end tag closes the most recently opened tag of that name, and every tag still open inside it; an end tag with no matching open tag is ignored. Tags still open at the end are closed then. Whitespace-only text becomes a single '\n' or ' ' string.
soup('a') is the same call as soup.find_all('a'): it walks every tag in document order and keeps those whose name matches and whose attributes pass the filter (a class_ value matches any one class). find() stops at the first match and returns None when nothing matches.
Each result is a Tag: .get('href', None) returns the value or None, tag['href'] raises KeyError when it is missing, .attrs is the attribute dict, .contents the list of children, .get_text() all the text inside, .string the single string child (or None).

Default run (69 steps): Raw HTML: 356 characters of plain text. BeautifulSoup(html, 'html.parser') will turn it into a tree. … [tag.get('href', None) for tag in soup('a')] → ['https://www.py4e.com/', '/lessons', '#top', None]

Simplified: A from-scratch port of Python 3.12's html.parser and BeautifulSoup 4.15's html.parser tree builder, checked against the real libraries on 330+ documents and 12,000+ queries. Documents are limited to 1,500 characters; named character references are limited to about 50 common names (others stay literal text); searches take one tag name and at most one attribute filter (no regular expressions, lists, functions, string= or CSS select()). Other parsers (lxml, html5lib) build different trees for broken HTML.

Educational simulation

Loading the simulation…

From Page to Link List

A web page displays clickable links, but a scraper sees those links as HTML anchor tags. An anchor tag stores its destination in an href attribute. BeautifulSoup can parse downloaded HTML, find every anchor tag, and retrieve each href value. When something goes wrong, the reliable approach is to trace the data from downloaded bytes to the final printed values instead of guessing which step failed.

How does data move from raw HTML through BeautifulSoup parsing and anchor-tag selection to the final list of extracted URLs?

Tracing the Extraction Steps

1
urllib.request.urlopen() opens the address over the network.
2
read() obtains the downloaded HTML as bytes.
3
BeautifulSoup parses those bytes and builds an internal tree representing the HTML document.
4
soup('a') searches the parsed tree and returns all anchor tag elements.
5
tag.get('href', None) retrieves the href value from each anchor tag, or returns None when that attribute is absent.
6
print() sends each extracted value to the console.

Which href value belongs to each anchor tag, and what happens when an anchor has no href attribute?

The expression soup('a') searches the entire parsed document tree, not just one particular part of the page. It returns a list of all matching anchor tag elements. Each returned tag can contain attributes such as href. Calling tag.get('href', None) asks for the href attribute safely: an existing attribute produces its stored value, while a missing attribute produces None.

python

from bs4 import BeautifulSoup

html = b'<a href="https://example.com">Example</a><a>Missing destination</a>'
soup = BeautifulSoup(html, "html.parser")
tags = soup('a')

for tag in tags:
    href = tag.get('href', None)
    print(href)

from bs4 import BeautifulSoup

html = b'<a href="https://example.com">Example</a><a>Missing destination</a>'
soup = BeautifulSoup(html, "html.parser")
tags = soup('a')

for tag in tags:
    href = tag.get('href', None)
    print(href)

Output

https://example.com
None

What elements does soup('a') find in the parsed document, and how are multiple matching anchor tags returned?

Certificate Verification

When urllib.request.urlopen() opens a URL, Python checks the website's SSL/TLS certificate. The check is intended to verify that the connection is secure and that the connection is with the real server rather than an imposter. In some websites or network environments, this check can fail even when the website is legitimate.

python

import ssl
import urllib.request

context = ssl.create_default_context()
context.check_hostname = False
context.verify_mode = ssl.CERT_NONE

response = urllib.request.urlopen(url, context=context)
html = response.read()

import ssl
import urllib.request

context = ssl.create_default_context()
context.check_hostname = False
context.verify_mode = ssl.CERT_NONE

response = urllib.request.urlopen(url, context=context)
html = response.read()

What changes in the request flow when certificate verification succeeds, fails, or is explicitly configured?

Debugging the Data Flow

A link extractor can fail at several different points. The HTML might not have been downloaded or parsed correctly. The search might find fewer anchor tags than expected. Or the anchor tags might exist without href attributes. Add observations at each stage so the first unexpected value identifies the part of the pipeline that needs investigation.

Code walkthrough

html = response.read()
print("downloaded bytes:", len(html))

soup = BeautifulSoup(html, "html.parser")
print("soup created:", soup is not None)

tags = soup('a')
print("anchor tags:", len(tags))

for tag in tags:
    href = tag.get('href', None)
    print("href:", href)

1
First inspect len(html). A value of 0 indicates that the download produced no bytes.
2
Next inspect soup creation and the parsed document before investigating link values.
3
Then inspect len(tags). A value of 0 means that no anchor tags were found, either because the page has no links or because parsing did not work as expected.
4
Finally inspect each href. If every value is None, the anchor tags exist but do not have href attributes.

The length of html shows whether any downloaded bytes reached the parser.
The soup object represents the parsed HTML tree.
The length of tags shows how many anchor elements the search found.
Each href print shows the value retrieved from one anchor tag.

Observation	What it indicates	Where to investigate
len(html) is 0	The download produced no HTML bytes	The download step
len(tags) is 0	No anchor tags were found	The page content or parsing step
href is None for every tag	Anchor tags exist without href attributes	The HTML attributes
Some href values appear	The extraction reached the attribute values	The final output and expected values

Use observations at each stage to locate the first divergence in link extraction.

How does get('href', None) return a URL when href exists and None when it is missing?

Mistakes to Avoid

Common mistake
Accessing href directly instead of using get()
What it looks like
Treating every anchor tag as though it definitely has an href attribute
Why it fails
An anchor tag can exist without an href attribute.
Do this instead
Use tag.get('href', None) so a missing attribute produces None.
Common mistake
Forgetting to pass the SSL context
What it looks like
Creating an SSL context but calling urlopen(url) without context=context
Why it fails
The configured context is not used unless it is passed to urlopen().
Do this instead
Pass the context through the context parameter when opening the URL.
Common mistake
Debugging only the final printed list
What it looks like
Assuming extraction is broken without checking downloaded size or the number of tags found
Why it fails
The failure may have occurred during download, parsing, or tag selection rather than attribute retrieval.
Do this instead
Trace len(html), soup creation, len(tags), and each href value.
Common mistake
Assuming every href is a complete web address
What it looks like
Treating tutorial/index.html or #whatsnew/3.6.html as though they were absolute URLs
Why it fails
The program prints the exact href stored in each anchor tag, including relative paths and fragments.
Do this instead
Interpret the output as raw attribute values before deciding how to process them.

Practice the Trace

EASY

Suppose a parsed document contains three anchor tags. The first has href="/guide", the second has no href attribute, and the third has href="https://example.com/next". Predict the three values printed by a loop that uses tag.get('href', None). Then identify which debug value would tell you how many anchor tags were found.

Hints

The loop prints the stored href value for each tag in order.
A missing href attribute produces None.
The number of matching anchor tags is obtained with len(tags).

Your attempt (not saved)

What do you think happens?

What does the loop print for an anchor tag that has no href attribute?

An empty string
None
The visible anchor text
A certificate error

Reveal answer

Answer:

None

tag.get('href', None) returns the href value when the attribute exists and returns None when the attribute is absent.

Reliable Extraction

1
The extraction pipeline moves from downloaded HTML bytes to a BeautifulSoup tree, then to anchor tags, href values, and printed output.
2
soup('a') searches the complete parsed document tree and returns all anchor tag elements it finds.
3
tag.get('href', None) safely retrieves an href value and returns None when the attribute is missing.
4
Certificate verification can be explicitly disabled with an SSL context, but doing so has security implications.
5
Debugging is most effective when you inspect download size, soup creation, tag count, and extracted href values in sequence.

Key Takeaways

BeautifulSoup turns downloaded HTML bytes into a searchable document tree.
soup('a') returns all anchor tags in that parsed tree.
tag.get('href', None) returns an href value or None when href is absent.
SSL/TLS certificate handling affects whether urlopen() can complete the request.
Tracing each intermediate value reveals where link extraction diverges from expectations.

More concepts starting with H