Web Scraping Ethics and Best Practices

For this topic, a practical best practice is to choose a parser that matches the data you actually receive.

Programming

The Real-World HTML Problem

HTML resembles XML because both use tags, attributes, and hierarchical nesting. Their treatment of imperfect input is different, however. XML requires a strict, well-formed structure, while production HTML commonly contains missing closing tags, unclosed attributes, and nesting violations. A parser designed to enforce XML rules may reject the entire page when it encounters one of these problems.

Strict and Tolerant Parsing

A strict parser enforces the rules of its specification. Missing closing tags, improper nesting, and malformed attributes can cause immediate failure. A tolerant parser takes a different approach: it makes reasonable assumptions about the author's intent, infers where a missing tag should close, and does its best to extract malformed attribute values. BeautifulSoup is useful for web scraping because it tolerates these common imperfections instead of stopping at the first malformed part.

What happens when malformed HTML is processed by a strict XML parser instead of BeautifulSoup?

Strict XML parsing	BeautifulSoup parsing
Requires well-formed structure	Handles real-world malformed HTML
Can stop when a rule is violated	Tolerates malformations and continues building a structure
Rejects improper nesting or malformed attributes	Makes assumptions about intended structure
Produces failure for rejected input	Produces a navigable tree when possible

The central difference is how each approach responds to imperfect markup.

Building the BeautifulSoup Object

BeautifulSoup does not leave the HTML as an undifferentiated string. The workflow transforms raw HTML text into a BeautifulSoup object. You provide the HTML string and a parser specification to the BeautifulSoup constructor. BeautifulSoup reads the input, tolerates malformations, and constructs an internal tree. The resulting object is navigable and queryable: elements can be reached by tag name, attributes, or position in the tree.

How does raw HTML text move through BeautifulSoup and become a structure with elements, attributes, and relationships?

Key point

The important transformation is from text to relationships. After parsing, each HTML element becomes a node in the internal structure, so the document can be explored rather than treated only as a sequence of characters.

The HTML Parent-Child Tree

HTML elements form a hierarchy. The html element is typically the root, with head and body as children. Within body, elements such as divs, paragraphs, and links can contain other nested elements. This nesting creates parent-child relationships. Elements can also have sibling relationships when they share the same parent. BeautifulSoup exposes these relationships so you can move through parent nodes, child nodes, and sibling nodes while locating data.

How does nesting turn HTML elements into a navigable parent-child hierarchy?

Worked example

Tracing a Nested Element

Suppose a parsed document contains an html root, a body element, a div inside body, and a paragraph inside the div. How can the relationships be described?

1
Start at html
Treat html as the typical root of the document tree.
2
Move to body
body is a child of html and contains the document's visible page structure.
3
Move to div
The div is a child of body and acts as a container for nested elements.
4
Move to paragraph
The paragraph is a child of div. Its parent is div, and other elements inside the same div may be its siblings.

Result

The parsed object represents nesting as relationships, allowing traversal from parent to child and between sibling elements.

Installation and Import

1
Download and install BeautifulSoup before using it.
2
Use pip, Python's package manager, to install the library from the Python Package Index.
3
Import BeautifulSoup into the Python script after installation.
4
Provide HTML text and a parser specification to the BeautifulSoup constructor.
5
Work with the resulting object as a navigable and queryable HTML structure.

Mistakes in Parser Selection

Common mistake
Assuming that HTML must satisfy XML's strict well-formedness rules
What it looks like
Treating missing closing tags or improper nesting as proof that the entire web page cannot be processed
Why it fails
Real-world HTML often contains these imperfections, and a strict XML parser may reject the whole page because of them.
Do this instead
Use a tolerant HTML-oriented approach such as BeautifulSoup when the input is ordinary, potentially malformed web HTML.
Common mistake
Expecting the parser to return only the original text
What it looks like
Thinking of parsing as reading a string without creating relationships between elements
Why it fails
BeautifulSoup builds an internal tree in which elements become nodes connected by nesting and sibling relationships.
Do this instead
Think of the result as a navigable and queryable BeautifulSoup object.
Common mistake
Ignoring the parser specification passed to the constructor
What it looks like
Describing the workflow as if BeautifulSoup receives HTML alone
Why it fails
The documented workflow passes both the HTML string and a parser specification to the constructor.
Do this instead
Include both inputs when reasoning about how the parsing process begins.
Common mistake
Looking only for elements by visible position
What it looks like
Overlooking the fact that a target element can be located through its tag name, attributes, or tree position
Why it fails
The parsed structure supports several ways to access elements.
Do this instead
Use the tree's names, attributes, and relationships as navigation routes.

Practice the Transformation

MEDIUM

Explain, in order, what happens when malformed HTML is supplied to BeautifulSoup. Your explanation should include the raw HTML text, the parser specification, tolerance of malformations, construction of the internal tree, and the resulting navigable object.

Hints

Begin with the two inputs supplied to the constructor.
Describe what tolerant parsing does when the markup is imperfect.
End by explaining what relationships become available after the tree is built.

Your attempt (not saved)

What do you think happens?

A web page contains a missing closing tag. Which result best matches the two parsing approaches?

Both parsers always reject the page
A strict XML parser may reject the page, while BeautifulSoup tolerates the problem and builds a tree
BeautifulSoup returns the original text without structure
A strict XML parser automatically repairs every error

Reveal answer

Answer:

A strict XML parser may reject the page, while BeautifulSoup tolerates the problem and builds a tree

Strict parsing enforces well-formed structure. BeautifulSoup is designed to tolerate malformed HTML and construct a navigable internal tree.

Key Takeaways

1
Real-world HTML often contains missing closing tags, unclosed attributes, and improper nesting.
2
Strict XML parsers can reject malformed HTML, while BeautifulSoup tolerates many such imperfections.
3
BeautifulSoup transforms raw HTML text into a navigable object by constructing an internal tree.
4
The tree represents parent, child, and sibling relationships among HTML elements.
5
BeautifulSoup must be installed through pip and imported before it can be used in a Python script.

At a glance

BeautifulSoup is designed for parsing real-world HTML that may not satisfy strict XML rules.
Tolerant parsing makes assumptions about malformed input instead of immediately rejecting the entire document.
Parsing produces a tree-based BeautifulSoup object rather than leaving the HTML as plain text.
Tree relationships make elements accessible through parents, children, siblings, tag names, attributes, and position.
The basic setup consists of installing BeautifulSoup with pip, importing it, and passing HTML plus a parser specification to its constructor.

More concepts starting with W