Navigating BeautifulSoup Trees with find() and find_all()

BeautifulSoup transforms raw HTML strings into searchable tree-structured objects that mirror the document's logical hierarchy.

Programming

Interactive lab

Try it: Parsing HTML with BeautifulSoup

How BeautifulSoup(html, 'html.parser') turns raw HTML text into a tree of Tag objects, tolerating unclosed and mis-nested tags, and how soup('a'), find() and find_all() with class_ / attrs filters pick tags out of it so .get('href', None), tag['href'], .attrs, .contents, .get_text() and .string can extract their data.

How it works

html.parser reads the text left to right and reports start tags (with their attributes), end tags, text, comments and declarations.
Each start tag becomes a Tag inside the tag that is currently open; void elements such as <br> and <img> are closed at once; class and rel values are split into lists.
An end tag closes the most recently opened tag of that name, and every tag still open inside it; an end tag with no matching open tag is ignored. Tags still open at the end are closed then. Whitespace-only text becomes a single '\n' or ' ' string.
soup('a') is the same call as soup.find_all('a'): it walks every tag in document order and keeps those whose name matches and whose attributes pass the filter (a class_ value matches any one class). find() stops at the first match and returns None when nothing matches.
Each result is a Tag: .get('href', None) returns the value or None, tag['href'] raises KeyError when it is missing, .attrs is the attribute dict, .contents the list of children, .get_text() all the text inside, .string the single string child (or None).

Default run (69 steps): Raw HTML: 356 characters of plain text. BeautifulSoup(html, 'html.parser') will turn it into a tree. … [tag.get('href', None) for tag in soup('a')] → ['https://www.py4e.com/', '/lessons', '#top', None]

Simplified: A from-scratch port of Python 3.12's html.parser and BeautifulSoup 4.15's html.parser tree builder, checked against the real libraries on 330+ documents and 12,000+ queries. Documents are limited to 1,500 characters; named character references are limited to about 50 common names (others stay literal text); searches take one tag name and at most one attribute filter (no regular expressions, lists, functions, string= or CSS select()). Other parsers (lxml, html5lib) build different trees for broken HTML.

Educational simulation

Loading the simulation…

From HTML Text to a Searchable Tree

When urllib fetches a web page, the result is raw HTML: a long string of characters. That string is difficult to navigate directly. BeautifulSoup parses the HTML and transforms it into a tree-structured object that mirrors the document's logical hierarchy. Once the soup object exists, your program can search for tags and extract their attributes.

What does raw HTML become after BeautifulSoup parses it, and how are elements nested inside the resulting object?

Building the Soup Object

The central workflow has three stages. First, fetch the page so that you have raw HTML. Next, pass that HTML to BeautifulSoup together with a parser specification. Finally, search the resulting soup object for the tags you need. The parser='html.parser' argument selects Python's built-in HTML parser rather than an external parser.

python

from bs4 import BeautifulSoup

html = '<html><body><h1>Links</h1><a href="/about">About</a></body></html>'
soup = BeautifulSoup(html, 'html.parser')

from bs4 import BeautifulSoup

html = '<html><body><h1>Links</h1><a href="/about">About</a></body></html>'
soup = BeautifulSoup(html, 'html.parser')

After the final line runs, soup represents the entire HTML document as a navigable BeautifulSoup object. The original HTML was a flat string for the program; the parsed result provides a tree-like structure that can be searched by tag name.

Choosing One Match or Every Match

Once the document has been parsed, navigation becomes a search decision. Use find() when the task calls for one matching element. Use find_all() when the task calls for every matching element of a type. The broader BeautifulSoup pattern described in the source is soup('a'), which returns a list containing all anchor tags. Thus, a page with several links requires a collection-oriented search when you want to inspect every link.

What is the difference between a search for one matching element and a search for all matching elements?

Code walkthrough

first_link = soup.find('a')
all_links = soup.find_all('a')

print(first_link)
print(all_links)

1
The first line searches the parsed tree for one anchor tag.
2
The second line searches the parsed tree for all anchor tags.
3
The print statements let you inspect the different result shapes.

find('a') asks for one matching anchor element.
find_all('a') asks for all matching anchor elements.
The first result is one tag; the second result is a collection of tags.

The source-supported shorthand soup('a') also returns all anchor tags. In practical terms, the important distinction is whether the next operation expects one tag or a collection of tags. If you are collecting every link, iterate over the collection and process each tag separately.

Following an Anchor to Its href

An anchor tag is an element in the parsed tree, while href is an attribute attached to that element. After finding anchor tags, call tag.get('href', None) to retrieve the href value. The second argument is the fallback value. If an anchor has no href attribute, get() returns None instead of raising an error.

How does data move from an anchor tag in the HTML tree to the href value returned by get()?

Worked example

Collecting Link Destinations Safely

Given a parsed page, inspect every anchor tag and retrieve its href value without failing when an anchor lacks that attribute.

1
Find the anchors
Search the soup object for all anchor tags. The result is a collection of matching tags.
2
Inspect each tag
Process one anchor tag at a time inside a loop.
3
Read href safely
Call tag.get('href', None). A present href produces its value; a missing href produces None.

Result

The program can report each link value and can also reveal which anchor tags do not contain href.

python

for tag in soup.find_all('a'):
    href = tag.get('href', None)
    print(href)

for tag in soup.find_all('a'):
    href = tag.get('href', None)
    print(href)

Access pattern	When the attribute exists	When the attribute is missing
tag.get('href', None)	Returns the href value	Returns None
tag['href']	Returns the href value	Raises KeyError

get() with a default is safer for incomplete or malformed HTML.

Recovering from SSL Certificate Errors

The parsing steps begin only after the page has been fetched. A request made with urllib can encounter an SSL certificate error before BeautifulSoup receives any HTML. The error-handling part of the workflow is therefore placed around page retrieval, not around tag searching.

What control flow occurs when page retrieval encounters an SSL certificate error, and where can the error be handled?

The source describes handling this situation by creating and configuring an SSL context that bypasses verification when necessary. That configuration disables hostname checking and certificate verification. It may be useful in educational examples or testing environments, but production code should verify certificates so that connections remain secure.

Mistakes That Break Extraction

Common mistake
Trying to search raw HTML as though it were already a BeautifulSoup tree.
What it looks like
Searching before passing the fetched HTML to BeautifulSoup.
Why it fails
The fetched page is still only a long string of characters.
Do this instead
Create a BeautifulSoup object with the HTML and a parser before searching for tags.
Common mistake
Using a one-result search when the task requires every matching anchor.
What it looks like
Using find('a') when the goal is to collect all links.
Why it fails
A one-result search does not represent the complete collection needed for the task.
Do this instead
Use find_all('a') or the source-supported soup('a') pattern when processing every anchor.
Common mistake
Assuming every anchor has an href attribute.
What it looks like
Using tag['href'] for every anchor.
Why it fails
An anchor without href causes bracket notation to raise KeyError.
Do this instead
Use tag.get('href', None) so a missing attribute produces None.
Common mistake
Treating an SSL certificate error as a BeautifulSoup parsing problem.
What it looks like
Trying to change tag-searching code when urllib cannot fetch the page.
Why it fails
The failure occurs during page retrieval, before BeautifulSoup receives HTML.
Do this instead
Handle the retrieval issue with an appropriately configured SSL context, while remembering the production security warning.

Practice the Retrieval Pipeline

EASY

Imagine that a fetched page has already been parsed into soup. Write the extraction logic that searches for every anchor tag, retrieves each tag's href value with a default of None, and prints the result. Then explain why get() is safer than bracket notation for this task.

Hints

Use a collection-oriented search for the anchor tags.
Process each returned tag in a loop.
Pass both 'href' and None to get().

Your attempt (not saved)

What do you think happens?

A page contains three anchor tags, but one of them has no href attribute. What should tag.get('href', None) produce for that anchor?

The entire HTML document
None
A KeyError
The tag name

Reveal answer

Answer:

None

The second argument to get() is the default value. It is returned when the requested attribute is missing.

Workflow Summary

1
Fetch the page with urllib and obtain raw HTML text.
2
If retrieval encounters an SSL certificate error, handle it at the request stage with an SSL context when necessary.
3
Pass the HTML to BeautifulSoup with a parser such as html.parser.
4
Use find() when you need one matching element and find_all() when you need all matching elements.
5
For each anchor tag, use tag.get('href', None) to retrieve href safely.
6
Treat None as an indication that the particular tag has no href attribute.

At a glance

BeautifulSoup transforms raw HTML text into a searchable tree that mirrors the document's logical hierarchy.
find() is appropriate for retrieving one matching element, while find_all() is appropriate for retrieving all matching elements.
Anchor attributes can be read safely with tag.get('href', None), which returns None when href is missing instead of raising KeyError.
SSL certificate handling belongs around the urllib retrieval step, before BeautifulSoup parses the HTML.
Disabling hostname checking and certificate verification may help in educational or testing environments, but production code should verify certificates.

Related concepts

More concepts starting with N