Concepts / Finding and Selecting Tags with BeautifulSoup

Finding and Selecting Tags with BeautifulSoup

Tag inspection in BeautifulSoup means accessing the individual components of a Tag object: its attributes (via get() or attrs), its text content (via contents or get_text()), and its child elements (via contents).

  • Programming
Interactive lab

Try it: Parsing HTML with BeautifulSoup

How BeautifulSoup(html, 'html.parser') turns raw HTML text into a tree of Tag objects, tolerating unclosed and mis-nested tags, and how soup('a'), find() and find_all() with class_ / attrs filters pick tags out of it so .get('href', None), tag['href'], .attrs, .contents, .get_text() and .string can extract their data.

How it works

  1. html.parser reads the text left to right and reports start tags (with their attributes), end tags, text, comments and declarations.
  2. Each start tag becomes a Tag inside the tag that is currently open; void elements such as <br> and <img> are closed at once; class and rel values are split into lists.
  3. An end tag closes the most recently opened tag of that name, and every tag still open inside it; an end tag with no matching open tag is ignored. Tags still open at the end are closed then. Whitespace-only text becomes a single '\n' or ' ' string.
  4. soup('a') is the same call as soup.find_all('a'): it walks every tag in document order and keeps those whose name matches and whose attributes pass the filter (a class_ value matches any one class). find() stops at the first match and returns None when nothing matches.
  5. Each result is a Tag: .get('href', None) returns the value or None, tag['href'] raises KeyError when it is missing, .attrs is the attribute dict, .contents the list of children, .get_text() all the text inside, .string the single string child (or None).

Default run (69 steps): Raw HTML: 356 characters of plain text. BeautifulSoup(html, 'html.parser') will turn it into a tree. … [tag.get('href', None) for tag in soup('a')] → ['https://www.py4e.com/', '/lessons', '#top', None]

Simplified: A from-scratch port of Python 3.12's html.parser and BeautifulSoup 4.15's html.parser tree builder, checked against the real libraries on 330+ documents and 12,000+ queries. Documents are limited to 1,500 characters; named character references are limited to about 50 common names (others stay literal text); searches take one tag name and at most one attribute filter (no regular expressions, lists, functions, string= or CSS select()). Other parsers (lxml, html5lib) build different trees for broken HTML.

Educational simulation

Loading the simulation…

From HTML Text to Inspectable Tags

BeautifulSoup does more than preserve HTML as a string. When raw HTML is parsed, BeautifulSoup builds a tree and returns Tag objects for elements in that tree. An anchor element, written with the a tag, therefore becomes a structured object whose name, attributes, inner content, and position among other elements can be inspected separately.

Key point

The central idea is to stop treating an anchor as opaque HTML text. Inspect the Tag object to retrieve the href value, the text a user sees, the complete attribute collection, or nested child elements.

The Parsed Document Tree

Parsing comes first. The html.parser reads the HTML string, recognizes tag boundaries, extracts attributes, and builds a navigable tree. Only after this tree exists can you find individual anchor tags and inspect their parts.

Parsed documentHTML treebodyahref=/homeahref=#detailscontainscontainscontains
How does BeautifulSoup search the parsed document and return each matching anchor tag?
python

The find_all("a") operation selects every anchor tag in the parsed document. The result is a collection of matching Tag objects, so each item can then be inspected independently.

Inside an Anchor Tag

A Tag maintains several related parts. Its tag name identifies the element, its attributes hold key-value information such as href, class, or id, and its inner content contains text and possibly nested elements. The Tag also participates in the document tree through parent, sibling, and child relationships.

atag namehrefattributeclassattributeRead moreinner textspanchild elementhashascontainscontains
What does an anchor Tag contain, and how are its tag name, attributes, text content, and child elements connected?

Think of the Tag as a small inspection surface rather than a single value. The same object can answer different questions: What element is this? What attributes does it have? What child nodes are inside it? What text would a user read?

QuestionTag part or operationWhat it provides
What element is this?Tag nameThe element name, such as a
What URL is attached?get('href', None)The href value, or None when href is absent
What attributes exist?attrsThe tag's attributes
What nodes are inside?contentsA list containing text and child elements
What visible text is present?get_text().strip()Text with surrounding whitespace removed

Common inspection questions and the corresponding BeautifulSoup access pattern

Reading Attributes and Text

Worked example

Inspecting one anchor

Given an anchor Tag created from HTML, retrieve its href, visible text, and attributes without failing when href is missing.

  1. 1
    Retrieve the URL safely

    Use tag.get('href', None). The get method returns the attribute value when href exists and returns the supplied default when it does not.

  2. 2
    Clean the visible text

    Use tag.get_text().strip(). The strip operation removes surrounding whitespace that may have come from formatting in the original HTML.

  3. 3
    Inspect all attributes

    Use tag.attrs to examine the complete attribute collection. The source material describes this collection as attribute entries represented as tuples inside a list.

  4. 4
    Inspect the inner structure

    Use tag.contents when you need the individual text and child-element items between the opening and closing anchor tags.

Result

One Tag object can provide a safe URL value, cleaned visible text, all attributes, and the list of inner content items.

python
Output
url: /guide
text: Read the guide
attributes: href and class entries
contents: whitespace text, followed by Read the guide text, followed by whitespace text

The important distinction is that contents preserves the inner structure. Its list may include NavigableString text nodes, whitespace, and nested Tag objects. get_text() instead gives you the rendered text from the tag's content; applying strip() is useful when the HTML contains indentation or line breaks.

URL Forms in Real Pages

An anchor's href is not always a complete web address. Real pages can contain absolute URLs, relative paths, and in-page references. Inspect the href value before deciding how your program should use it, and filter and validate URLs before using them.

https://example.com/pageabsolute URL/guiderelative path#detailsin-page referencedifferent href formdifferent href form
How can you distinguish absolute URLs, relative paths, and in-page references by inspecting an anchor's href value?
Href formExampleInspection implication
Absolute URLhttps://example.com/pageThe href contains a complete web address
Relative path/guideThe href identifies a path relative to the page context
In-page reference#detailsThe href points to a location within the page

The three URL forms emphasized in the source material

Best practice

Do not assume every href is ready to fetch or use as an external address. First retrieve it safely, inspect which form it has, and filter and validate it for the task your program is performing.

Mistakes During Inspection

  • Common mistake

    Assuming every anchor has an href attribute.

    What it looks like
    url = tag['href']
    Why it fails

    Direct dictionary-style access raises a KeyError when the attribute is missing.

    Do this instead

    Use tag.get('href', None) when the attribute may not exist.

  • Common mistake

    Using contents as if it were already clean visible text.

    What it looks like
    text = tag.contents
    Why it fails

    contents is a list and may include whitespace, text nodes, and child tags.

    Do this instead

    Use tag.get_text().strip() when the goal is clean text extraction.

  • Common mistake

    Treating an href as if it always were an absolute URL.

    What it looks like

    Using /guide and #details exactly like complete web addresses.

    Why it fails

    Real web pages contain absolute URLs, relative paths, and in-page references.

    Do this instead

    Inspect, filter, and validate the href before using it.

  • Common mistake

    Treating a Tag as only its printed HTML.

    What it looks like

    Examining only the string representation of an anchor.

    Why it fails

    The Tag separately holds its name, attributes, inner content, and document-tree relationships.

    Do this instead

    Inspect the relevant Tag properties and methods directly.

A Small Inspection Routine

Code walkthrough
from bs4 import BeautifulSoup

html = """
<a href="https://example.com">External page</a>
<a href="/guide">Guide</a>
<a href="#details">Details</a>
<a>Missing URL</a>
"""

soup = BeautifulSoup(html, "html.parser")

for tag in soup.find_all("a"):
    href = tag.get("href", None)
    text = tag.get_text().strip()
    print(href, text)
  1. 1
    The parser creates a document tree from the HTML.
  2. 2
    The loop visits every anchor Tag returned by find_all('a').
  3. 3
    The href value may be an absolute URL, a relative path, an in-page reference, or None.
  4. 4
    The text value is obtained separately from the URL and surrounding whitespace is removed.
  • BeautifulSoup parses the HTML string with html.parser.
  • find_all('a') returns each anchor Tag.
  • get retrieves href safely, including for the anchor with no href.
  • get_text().strip() extracts clean visible text.
MEDIUM

For each anchor in the routine, classify the href as an absolute URL, relative path, in-page reference, or missing attribute. Then explain why get('href', None) is safer than direct dictionary-style access.

Hints
  • Look at the beginning of each href value.
  • A missing href produces the default value None.
  • Use get_text().strip() for the visible text.

Inspection Checklist

  1. 1
    Parse the HTML with BeautifulSoup and html.parser.
  2. 2
    Use find_all('a') to retrieve all anchor Tag objects.
  3. 3
    Use get('href', None) for safe URL retrieval.
  4. 4
    Use get_text().strip() for clean visible text.
  5. 5
    Use contents when you need the individual inner text and child-element items.
  6. 6
    Use attrs to inspect the complete attribute collection.
  7. 7
    Classify, filter, and validate href values because pages contain multiple URL forms.

Key Takeaways

  • BeautifulSoup turns parsed HTML elements into structured Tag objects.
  • find_all('a') retrieves the anchor tags in a parsed document.
  • A Tag exposes attributes, inner content, child elements, and clean text through separate access patterns.
  • Use get('href', None) for safe attribute access and get_text().strip() for clean text.
  • Inspect and validate href values because anchors may contain absolute URLs, relative paths, or in-page references.

Related Topics

Related concepts

More concepts starting with F