Error Handling in Web Scraping

Web scraping begins with data that is difficult to search directly: when urllib fetches a web page, it returns raw HTML as one long string of characters.

Programming

Interactive lab

Try it: Parsing HTML with BeautifulSoup

How BeautifulSoup(html, 'html.parser') turns raw HTML text into a tree of Tag objects, tolerating unclosed and mis-nested tags, and how soup('a'), find() and find_all() with class_ / attrs filters pick tags out of it so .get('href', None), tag['href'], .attrs, .contents, .get_text() and .string can extract their data.

How it works

html.parser reads the text left to right and reports start tags (with their attributes), end tags, text, comments and declarations.
Each start tag becomes a Tag inside the tag that is currently open; void elements such as <br> and <img> are closed at once; class and rel values are split into lists.
An end tag closes the most recently opened tag of that name, and every tag still open inside it; an end tag with no matching open tag is ignored. Tags still open at the end are closed then. Whitespace-only text becomes a single '\n' or ' ' string.
soup('a') is the same call as soup.find_all('a'): it walks every tag in document order and keeps those whose name matches and whose attributes pass the filter (a class_ value matches any one class). find() stops at the first match and returns None when nothing matches.
Each result is a Tag: .get('href', None) returns the value or None, tag['href'] raises KeyError when it is missing, .attrs is the attribute dict, .contents the list of children, .get_text() all the text inside, .string the single string child (or None).

Default run (69 steps): Raw HTML: 356 characters of plain text. BeautifulSoup(html, 'html.parser') will turn it into a tree. … [tag.get('href', None) for tag in soup('a')] → ['https://www.py4e.com/', '/lessons', '#top', None]

Simplified: A from-scratch port of Python 3.12's html.parser and BeautifulSoup 4.15's html.parser tree builder, checked against the real libraries on 330+ documents and 12,000+ queries. Documents are limited to 1,500 characters; named character references are limited to about 50 common names (others stay literal text); searches take one tag name and at most one attribute filter (no regular expressions, lists, functions, string= or CSS select()). Other parsers (lxml, html5lib) build different trees for broken HTML.

Educational simulation

Loading the simulation…

From Raw Text to Searchable HTML

Web scraping begins with data that is difficult to search directly: when urllib fetches a web page, it returns raw HTML as one long string of characters. The useful information is present, but it is not yet organized for convenient programmatic navigation. BeautifulSoup transforms that string into a tree-structured object that mirrors the document's logical hierarchy. Once this transformation is complete, you can search for tags and retrieve their attributes.

How does a flat HTML string become a searchable structure in which elements contain other elements?

Preparing BeautifulSoup

Before extracting tags or attributes, BeautifulSoup must be installed and available as a dependency. The scraping workflow then has three main stages: fetch the page with urllib, pass the returned HTML to BeautifulSoup together with a parser specification, and search the resulting object for the tags you need. The parser='html.parser' argument tells BeautifulSoup to use Python's built-in HTML parser rather than an external parser.

Install BeautifulSoup before attempting to parse HTML.
Fetch the web page so that urllib provides the raw HTML string.
Create a BeautifulSoup object using the raw HTML and the parser specification.
Search the parsed object for the tags and attributes required by the task.

Finding Anchor Tags

Worked example

Locating every link in a parsed document

Suppose a page has already been fetched and parsed into a BeautifulSoup object named soup. How can you retrieve its anchor tags?

1
Search by tag name
Call soup('a'). The string 'a' identifies the anchor tag, and the call returns a list containing all anchor tags found in the parsed document.
2
Inspect the returned collection
Each item in the returned list is an anchor tag represented inside the BeautifulSoup tree. These tag objects can then be examined for attributes.
3
Continue to attribute extraction
For each anchor tag, request the href attribute with get() rather than assuming every anchor has that attribute.

Result

The parsed document has been narrowed from the entire HTML tree to a list of anchor tags, ready for safe href extraction.

How are anchor tags located, and how does each href value connect to its anchor tag?

The expression soup('a') searches the parsed document for a tag named a. It returns a list of all matching anchor tags, not just the first one. This is useful when a page contains multiple links and you need to process them as a group.

Reading Attributes Safely

After retrieving an anchor tag, use tag.get('href', None) to request its href attribute. The first argument names the attribute. The second argument is the default value returned when that attribute is missing. With None as the default, the scraper can continue and the result shows that the particular anchor did not provide an href value.

What happens when href exists, when it is missing, and when bracket notation is used instead?

Access pattern	Attribute exists	Attribute is missing
tag.get('href', None)	Returns the href value	Returns None
tag['href']	Returns the href value	Raises a KeyError

The default supplied to get() allows missing href attributes to be handled without a KeyError.

In the real world

Real-world HTML may be malformed or incomplete. An anchor tag can exist without an href attribute. Using get() makes that condition visible through the default value while allowing the rest of the scraping process to continue.

Managing SSL Verification Failures

Fetching a page can fail because of an SSL certificate error. The source describes handling this situation by creating and configuring an SSL context that bypasses verification when necessary. This configuration disables hostname checking and certificate verification, allowing the fetch to proceed in educational examples or testing environments.

What happens in the request flow when certificate verification fails, and where does the configured SSL context change the path?

Mistakes Beginners Make

Common mistake
Treating fetched HTML as if it were already searchable.
What it looks like
Trying to retrieve tags directly from the raw string returned by urllib.
Why it fails
The fetched result is only a long string of characters from a programmatic perspective.
Do this instead
Pass the raw HTML to BeautifulSoup with the html.parser specification before searching for tags.
Common mistake
Searching for the wrong tag name.
What it looks like
Looking for links without using the anchor tag name a.
Why it fails
The soup(tag_name) pattern searches for the exact tag name supplied.
Do this instead
Use soup('a') when the goal is to retrieve anchor tags.
Common mistake
Assuming every anchor has an href attribute.
What it looks like
Using tag['href'] for every tag returned by soup('a').
Why it fails
An anchor without href causes bracket notation to raise a KeyError.
Do this instead
Use tag.get('href', None) so a missing attribute produces the default value instead.
Common mistake
Disabling SSL verification without considering the environment.
What it looks like
Using an SSL context that bypasses verification as a general production practice.
Why it fails
The configuration disables hostname checking and certificate verification.
Do this instead
Reserve the bypass for situations such as educational examples or testing environments, and verify certificates in production code.

Apply the Workflow

EASY

A page has been fetched with urllib and parsed into a BeautifulSoup object named soup. Describe the sequence you would use to locate every anchor tag and safely obtain each tag's href value. Include what result should be expected for an anchor that does not contain href.

Hints

Begin with the tag name used for anchor elements.
The search returns a list, so consider each returned tag.
Use get() with a default value for the attribute lookup.
A missing href should produce the chosen default rather than a KeyError.

Your attempt (not saved)

Worked example

Tracing the complete scraping path

Explain the state of the data at each stage of a scraper that must fetch a page, parse it, find links, and handle missing href attributes.

1
Fetch
urllib provides raw HTML as a long string of text.
2
Parse
BeautifulSoup transforms that string into a tree-structured object representing the document hierarchy.
3
Search
Calling soup('a') returns a list of all anchor tags in the parsed document.
4
Extract
Calling tag.get('href', None) returns each href value or None when the attribute is absent.
5
Recover from SSL failure
If certificate verification prevents fetching in an educational or testing environment, a configured SSL context can bypass verification so the workflow can continue.

Result

The complete pattern is fetch raw HTML, parse it, search for tags, safely extract attributes, and handle SSL verification failures according to the environment.

Key Takeaways

1
urllib returns fetched web pages as raw HTML strings; BeautifulSoup turns those strings into searchable tree-structured objects.
2
The parser='html.parser' specification uses Python's built-in HTML parser when creating the BeautifulSoup object.
3
The soup('a') pattern returns all anchor tags in the parsed document.
4
The tag.get('href', None) pattern safely returns an href value or None when href is missing, while bracket notation can raise a KeyError.
5
An SSL context can bypass hostname checking and certificate verification when necessary in educational or testing environments, but production code should verify certificates.

At a glance

Parse raw HTML with BeautifulSoup before attempting to search it.
Use soup('a') to retrieve all anchor tags from the parsed document.
Use tag.get('href', None) to handle missing href attributes without a KeyError.
Treat SSL verification bypassing as a limited option for educational or testing contexts, not as the production default.

More concepts starting with E