Extracting Data from Web Pages

A web page is presented as HTML, but extracting useful information requires more than treating that HTML as one large string.

Programming

Interactive lab

Try it: Parsing HTML with BeautifulSoup

How BeautifulSoup(html, 'html.parser') turns raw HTML text into a tree of Tag objects, tolerating unclosed and mis-nested tags, and how soup('a'), find() and find_all() with class_ / attrs filters pick tags out of it so .get('href', None), tag['href'], .attrs, .contents, .get_text() and .string can extract their data.

How it works

html.parser reads the text left to right and reports start tags (with their attributes), end tags, text, comments and declarations.
Each start tag becomes a Tag inside the tag that is currently open; void elements such as <br> and <img> are closed at once; class and rel values are split into lists.
An end tag closes the most recently opened tag of that name, and every tag still open inside it; an end tag with no matching open tag is ignored. Tags still open at the end are closed then. Whitespace-only text becomes a single '\n' or ' ' string.
soup('a') is the same call as soup.find_all('a'): it walks every tag in document order and keeps those whose name matches and whose attributes pass the filter (a class_ value matches any one class). find() stops at the first match and returns None when nothing matches.
Each result is a Tag: .get('href', None) returns the value or None, tag['href'] raises KeyError when it is missing, .attrs is the attribute dict, .contents the list of children, .get_text() all the text inside, .string the single string child (or None).

Default run (69 steps): Raw HTML: 356 characters of plain text. BeautifulSoup(html, 'html.parser') will turn it into a tree. … [tag.get('href', None) for tag in soup('a')] → ['https://www.py4e.com/', '/lessons', '#top', None]

Simplified: A from-scratch port of Python 3.12's html.parser and BeautifulSoup 4.15's html.parser tree builder, checked against the real libraries on 330+ documents and 12,000+ queries. Documents are limited to 1,500 characters; named character references are limited to about 50 common names (others stay literal text); searches take one tag name and at most one attribute filter (no regular expressions, lists, functions, string= or CSS select()). Other parsers (lxml, html5lib) build different trees for broken HTML.

Educational simulation

Loading the simulation…

From HTML to Inspectable Tags

A web page is presented as HTML, but extracting useful information requires more than treating that HTML as one large string. BeautifulSoup parses the HTML into a tree and returns Tag objects that can be inspected piece by piece. For an anchor tag, those pieces include the tag name, its attributes, its inner content, and its relationships with surrounding elements.

The parsing step comes first. The html.parser reads the HTML string, recognizes tag boundaries, extracts attributes, and builds a navigable tree. Once that tree exists, you can find individual anchor Tag objects and examine their parts instead of manually cutting up HTML text.

How does BeautifulSoup move from raw HTML to a collection of anchor tags?

Inside an Anchor Tag

A BeautifulSoup Tag is a structured object, not merely a printed copy of an HTML element. An anchor Tag has a name, such as a, attributes such as href, inner text or nested elements, and a position in the document tree. This structure lets you select only the information your task needs.

What does an anchor Tag contain, and how are its attributes, text, and child elements organized?

Key point

The contents list can contain both text and nested Tag objects. It can also contain whitespace from the original HTML, so contents is useful when you need the child-node structure, while get_text() is useful when you need readable text.

Choosing an Access Method

Tag inspection means choosing an access method that matches the part of the Tag you want. Use get() for a named attribute when the attribute might be absent. Use attrs to inspect the complete set of attributes. Use contents to inspect the immediate text and child nodes. Use get_text() when you want the displayed text in a cleaner form.

Which part of an anchor tag does each operation access?

Code walkthrough

href = tag.get('href', None)
attributes = tag.attrs
raw_parts = tag.contents
text = tag.get_text().strip()

1
The first line asks for href without raising an error when href is absent.
2
The second line inspects every attribute attached to the tag.
3
The third line preserves the tag's immediate contents for structural inspection.
4
The fourth line produces cleaner visible text by removing surrounding whitespace.

get() safely requests the href attribute and supplies None if it is missing.
attrs exposes the tag's complete attribute collection.
contents returns the immediate text and child nodes, including possible whitespace.
get_text().strip() extracts readable text and removes surrounding whitespace.

Following Every Anchor

After parsing a document, the extraction task has two levels. First, retrieve the collection of anchor tags. Then inspect each Tag separately and split its information into the URL, visible text, and other attributes. The loop is valuable because every anchor can have different content and different attributes.

How does one anchor Tag become separate URL, text, and attribute outputs?

python

for tag in soup.find_all('a'):
    href = tag.get('href', None)
    text = tag.get_text().strip()
    attributes = tag.attrs
    print(href, text, attributes)

for tag in soup.find_all('a'):
    href = tag.get('href', None)
    text = tag.get_text().strip()
    attributes = tag.attrs
    print(href, text, attributes)

Worked example

Inspecting One Anchor

Suppose an anchor Tag has an href attribute, visible text surrounded by whitespace, and additional attributes. Which operations retrieve the URL, readable text, and complete attribute collection?

1
Retrieve the URL
Call tag.get('href', None). This returns the href value when it exists and avoids an error when it does not.
2
Clean the visible text
Call tag.get_text().strip(). The get_text() method obtains the text, and strip() removes surrounding whitespace.
3
Inspect all attributes
Read tag.attrs to examine the attributes attached to the Tag.
4
Inspect structure when needed
Read tag.contents when the distinction between text nodes, whitespace, and child tags matters.

Result

One anchor can be represented as separate URL, text, and attribute results without treating the HTML element as opaque text.

What do you think happens?

If the HTML contains whitespace between an opening anchor tag and its text, what may appear in tag.contents?

Only the final visible text
The whitespace and the text
Only the href value
The complete document

Reveal answer

Answer:

The whitespace and the text

contents preserves the immediate contents of the tag, and that list may include whitespace as well as text and child Tag objects. get_text().strip() is the cleaner choice when readable text is wanted.

Recognizing URL Forms

The value returned from href is not guaranteed to have one uniform form. Real web pages contain absolute URLs, relative paths, and in-page references. These values should not be treated as interchangeable: inspect the form of each value, then filter and validate URLs before using them.

How does the href value differ for external links, site-relative links, and references within the current page?

Best practice

Treat href extraction and URL use as separate steps. First retrieve the attribute safely. Next identify whether the value is an absolute URL, a relative path, or an in-page reference. Finally filter and validate it for the purpose of your program.

Mistakes During Inspection

Common mistake
Assuming every anchor has an href attribute
What it looks like
Reading an attribute directly when href may be absent
Why it fails
Direct dictionary-style access raises a KeyError if the attribute does not exist.
Do this instead
Use tag.get('href', None) when the attribute may be missing.
Common mistake
Using contents as already-clean visible text
What it looks like
Printing tag.contents and expecting only the words a user sees
Why it fails
contents is a list and may include whitespace, text, and child Tag objects.
Do this instead
Use tag.get_text().strip() when clean text is required.
Common mistake
Treating attrs as the URL itself
What it looks like
Using the complete attribute collection as though it were the href value
Why it fails
attrs represents the tag's attributes, while href is one particular attribute.
Do this instead
Use tag.get('href', None) for the URL-like attribute and tag.attrs when inspecting all attributes.
Common mistake
Using every href value without checking its form
What it looks like
Passing absolute URLs, relative paths, and in-page references into the same later operation without inspection
Why it fails
Real pages contain different URL forms that should be filtered and validated before use.
Do this instead
Identify the URL form and validate it for the intended use.

Practice: Inspect a Link

MEDIUM

You are processing each anchor Tag in a parsed document. Write down which BeautifulSoup operation you would use for each task: obtain href safely when it may be absent, inspect every attribute, preserve text and child nodes, and obtain clean visible text. Then explain why an href value should be checked before your program uses it.

Hints

The safe attribute operation accepts an attribute name and a fallback value.
The complete attribute collection is exposed separately from one named attribute.
The structural list can include whitespace and nested Tag objects.
The text accessor can be combined with strip() for cleaner output.
Remember the three URL forms found on real web pages.

Your attempt (not saved)

Worked example

Practice Check

Match each task with its access method: one named attribute safely, all attributes, raw contents, or cleaned text.

1
One named attribute safely
Use tag.get('href', None).
2
All attributes
Use tag.attrs.
3
Raw contents
Use tag.contents.
4
Cleaned text
Use tag.get_text().strip().
5
Before using the URL
Inspect whether the value is absolute, relative, or an in-page reference, then filter and validate it.

Result

Each accessor answers a different inspection question, so selecting the method depends on whether you need an attribute, structure, or readable text.

Key Takeaways

1
BeautifulSoup parses HTML into a navigable tree containing structured Tag objects.
2
An anchor Tag exposes its attributes, contents, text, and document-tree relationships as inspectable parts.
3
Use tag.get('href', None) for safe href access, tag.attrs for the attribute collection, tag.contents for text and child nodes, and tag.get_text().strip() for cleaner text.
4
The contents list may preserve whitespace and nested Tag objects.
5
Absolute URLs, relative paths, and in-page references can all appear in real href values, so filter and validate URLs before using them.

At a glance

Parsing transforms raw HTML into a tree of inspectable BeautifulSoup Tag objects.
Anchor data is separated into attributes, contents, visible text, and document-tree relationships.
Safe href access prevents errors when an anchor lacks that attribute.
contents preserves structure and possible whitespace, while get_text().strip() targets clean readable text.
URL values require inspection, filtering, and validation because real pages contain multiple URL forms.

Related concepts

Parsing HTML with BeautifulSoup

More concepts starting with E