Analyzing Web Connectivity with PageRank
Analyzing web connectivity begins with collecting the connections between pages.
From Pages to Connections
Analyzing web connectivity begins with collecting the connections between pages. In this learning path, spider.py is the script that crawls websites and records links. As it visits pages, it extracts links from their HTML and uses those discoveries to find additional content. The collected pages and their connections are stored in spider.sqlite.
The central mechanism is discovery through HTML: spider.py visits a page, extracts links from that page, and uses the discovered links to continue finding content.
Tracing One Crawl
A Small Crawl Trace
Suppose spider.py visits a page whose HTML contains links to two other pages. What information becomes available after that visit?
- 1Visit
spider.pyprocesses the visited page. - 2Inspect HTML
The script extracts links from the HTML of the visited page.
- 3Record connections
The page and its connections are part of the crawl information recorded by
spider.py. - 4Continue discovery
The extracted links provide additional content that the crawl can continue to discover.
One visited page has contributed its discovered links to the growing record of web connectivity.
This trace separates two related ideas. Visiting a page gives the spider HTML to inspect. Extracting links turns that HTML into connectivity information. The crawl is therefore not just a list of pages; it also records how those pages are connected.
What do you think happens?
If a visited page contains links, what is the next important discovery step?
Reveal answer
Extract the links from the page HTML
The source describes link extraction from visited HTML pages as the way the spider discovers new content.
Persistent Crawl Records
spider.sqlite is the database used to persist the crawl. It stores all crawled pages and their connections. This gives the crawl a durable record of both the pages that have been crawled and the relationships discovered between them.
| Component | Role |
|---|---|
| spider.py | Crawls websites and records links. |
| Visited page HTML | Provides links from which new content is discovered. |
| spider.sqlite | Stores crawled pages and their connections. |
The main parts of the crawl and their responsibilities.
Script and Database
The script and the database have different roles. spider.py performs the crawling work and records what it discovers. spider.sqlite is the persistent place where the resulting page and connection information is stored. Thinking of them as a pair helps explain why crawl information can remain available beyond the moment when a page is visited.
When reasoning about a crawl, ask two separate questions: what did spider.py discover, and where is that discovery stored? The first question concerns the script; the second concerns spider.sqlite.
Resetting the Crawl
A new crawl requires clearing the existing persistent crawl record. To restart the crawling process, delete the spider.sqlite file. Deleting this file removes the stored database that contains the crawled pages and their connections, so the crawl can begin again without that existing record.
Common Crawl Mistakes
Common mistake Treating spider.py as the database
What it looks likeAssuming the script itself is where all crawled pages and connections are stored.
Why it failsspider.pyis the script used to crawl websites and record links, whilespider.sqliteis the database where crawled pages and connections are stored.Do this insteadKeep the roles separate: spider.py performs the crawl, and spider.sqlite stores the crawl record.
Common mistake Ignoring page HTML when explaining discovery
What it looks likeSaying that the spider discovers new content without considering the links on visited pages.
Why it failsThe spider discovers new content by extracting links from the HTML of visited pages.
Do this insteadDescribe discovery as a link-extraction step performed on visited HTML.
Common mistake Trying to restart without deleting spider.sqlite
What it looks likeRunning the crawler again while leaving the existing spider.sqlite file in place.
Why it failsThe required action for restarting a crawl is to delete
spider.sqlite.Do this insteadDelete spider.sqlite before starting the crawl again.
Recording a Link
Imagine that a visited page contains one link to another page. The important transformation is from page HTML to extracted link, and then from extracted link to recorded connection. spider.py carries out the crawling and recording, while spider.sqlite preserves the resulting information as part of the crawl database.
Practice Check
A learner says: spider.py finds pages, but spider.sqlite only stores the page currently being visited. They also want to restart the crawl and plan to run spider.py again without changing any files. Identify both errors and state the correct explanation or action.
Hints
- Recall what
spider.sqlitestores besides pages. - Separate the crawler script from its persistent database.
- Recall the source's required action for restarting a crawl.
Practice Answer
Correct the learner's two errors.
- 1Storage error
spider.sqlitestores all crawled pages and their connections, not only the page currently being visited. - 2Reset error
Running
spider.pyagain without deletingspider.sqlitedoes not perform the specified reset action. - 3Correct action
Delete
spider.sqliteto restart the crawl.
spider.py performs the crawl and records links; spider.sqlite stores the crawled pages and connections; deleting spider.sqlite resets the persistent crawl record.
Key Takeaways
- 1
spider.pycrawls websites and records links. - 2The spider discovers new content by extracting links from the HTML of visited pages.
- 3
spider.sqlitestores all crawled pages and their connections. - 4The crawler script and its database have separate roles.
- 5To restart a crawl, delete the
spider.sqlitefile.
At a glance
- A web spider builds connectivity information by extracting links from visited HTML pages.
spider.pyperforms the crawl and records discovered links.spider.sqlitepersistently stores crawled pages and their connections.- Deleting
spider.sqliteis the required way to restart the crawling process.