Concepts / Introduction to SQLite

Introduction to SQLite

The central relationship is simple: spider.py performs the crawl, while spider.sqlite stores the pages and connections found during that crawl.

  • Programming

A Crawl That Remembers

A web spider does more than visit one page. The spider.py script crawls websites, records links, and discovers new content by extracting links from the HTML of pages it visits. The spider.sqlite file provides persistent storage for the crawled pages and the connections between them.

Key point

The central relationship is simple: spider.py performs the crawl, while spider.sqlite stores the pages and connections found during that crawl.

Following the Crawl

spider.pycrawl and record linksVisited pageHTMLExtracted linksnew contentNext pagescontinued crawlvisitsHTML containsleads to
How does control move from the spider script to a visited page, link extraction, and additional crawling?

The crawl begins with spider.py. As the script visits a page, it examines that page's HTML. Links found in the HTML become discoveries that can lead the spider toward additional pages. This link-extraction step is the mechanism that allows the crawl to find new content rather than stopping at the first page.

Worked example

Tracing a Link Discovery

Imagine that spider.py visits a page whose HTML contains links to two other pages. What happens in the crawl?

  1. 1
    Visit

    spider.py visits the page as part of the website crawl.

  2. 2
    Inspect

    The spider examines the HTML of the visited page.

  3. 3
    Extract

    The spider discovers the links contained in that HTML.

  4. 4
    Continue

    The discovered links provide paths toward additional pages and new content.

Result

The crawl advances because links extracted from a visited page identify further content to crawl.

What spider.sqlite Stores

spider.pycrawl and recordspider.sqlitestored crawl dataCrawled pagespage recordsPage connectionslink relationshipsrecordsstoresstores
How do crawled pages and their connections move between spider.py and spider.sqlite during the crawling process?

spider.sqlite is the database where all crawled pages and their connections are stored. The database therefore records more than an isolated list of pages: it also stores how those pages are connected through discovered links.

Websitespages and HTMLspider.pycrawlerspider.sqlitepages and connectionsprovides pagesrecords crawl data
What information does spider.sqlite contain, and how is it connected to the spider script and the websites being crawled?

Resetting the Crawl

Before

spider.sqlite contains recorded crawled pages and their connections.

The existing crawl data remains stored in spider.sqlite.

After

spider.sqlite has been deleted.

The stored crawl record has been removed, so the crawl can be restarted.

To restart a crawl, delete the spider.sqlite file. Deleting this file removes the stored record of the crawled pages and their connections, which resets the persistent crawl state.

  • Common mistake

    Assuming that changing spider.py alone resets the crawl.

    What it looks like

    The script is changed, but spider.sqlite is left in place.

    Why it fails

    The crawl information is stored in spider.sqlite, so the existing database file still contains the recorded pages and connections.

    Do this instead

    Delete spider.sqlite when the goal is to restart the crawl.

  • Common mistake

    Treating spider.sqlite as the crawler itself.

    What it looks like

    Describing the database file as if it visits websites and extracts links.

    Why it fails

    spider.py is the script used to crawl websites and record links. spider.sqlite is where the crawled pages and their connections are stored.

    Do this instead

    Assign crawling and link recording to spider.py, and assign persistent storage to spider.sqlite.

  • Common mistake

    Ignoring the HTML link-extraction step.

    What it looks like

    Thinking the spider discovers new content without examining visited pages.

    Why it fails

    The spider discovers new content by extracting links from the HTML of visited pages.

    Do this instead

    Trace the crawl through a visited page's HTML and the links found there.

Check Your Understanding

EASY

A spider visits a page and finds links in its HTML. Explain which component performs the crawl, which component stores the crawled pages and connections, and what must be deleted to restart the crawl.

Hints
  • Separate the active script from the stored database file.
  • Use the sequence visited page, HTML, extracted links, and additional content.
  • The reset action concerns the file that stores crawl data.

What do you think happens?

If spider.sqlite is not deleted, has the crawl's stored state been reset?

  • Yes

  • No

Reveal answer
Answer:

No

The stored crawl data remains in spider.sqlite. To restart the crawl, the spider.sqlite file must be deleted.

Key Takeaways

  1. 1
    spider.py is the script that crawls websites and records links.
  2. 2
    The spider discovers new content by extracting links from the HTML of visited pages.
  3. 3
    spider.sqlite stores all crawled pages and their connections.
  4. 4
    Deleting spider.sqlite resets the stored crawl state and allows the crawl to restart.

At a glance

  • A web spider advances by visiting pages and extracting links from their HTML.
  • spider.py performs the crawling and records discovered links.
  • spider.sqlite stores crawled pages and the connections between them.
  • Deleting spider.sqlite is the required reset action for restarting the crawl.

Related Topics

Related concepts

More concepts starting with I