Introduction to PageRank: Measuring Page Importance
The database does not merely hold a list of URLs. It also preserves the direction of connections: one page is the source, and another page is the target.
Why Collection Comes First
PageRank-style analysis cannot begin with an empty view of the web. Before you can study which pages are central or influential, you must first collect pages and the links between them. A web crawler creates this starting dataset by visiting pages, recording their content, and noting which pages link to which other pages.
The spider.py program is the collection engine in this process. It takes a starting URL, fetches the page, extracts its links, stores the page and its outgoing links in spider.sqlite, and then continues with another unvisited link. The result is a stored graph: pages are the graph's entities, and links are the relationships connecting them. Later, this graph can be queried, analyzed with algorithms such as page rank, or visualized with D3.
From Web Page to Stored Records
The crawler handles two kinds of information for each processed page. First, it stores a page record containing a unique page ID, the URL, and the number of outgoing links. Second, it stores relationships for those links. A relationship identifies the source page by its ID and the target page by its ID.
The database does not merely hold a list of URLs. It also preserves the direction of connections: one page is the source, and another page is the target.
Following the Unvisited Queue
After fetching a page, spider.py extracts its links and adds the discovered relationships to the database. The crawler then selects another page that has not yet been visited. On a restart, it chooses a random unvisited page from its queue. The word systematic does not mean that every link is followed in a fixed order; it means that the crawler repeatedly applies the same discovery-and-storage process while respecting its crawl limit and its record of visited pages.
The Database Graph
The local SQLite file contains at least two tables: one for pages and one for links. The pages table represents the stored pages. The links table represents directed relationships between those pages. This separation allows later queries such as finding every page that links to a target, finding every page linked from a source, or calculating network statistics.
Suppose a stored page A links to pages B and C. The database keeps a page record for A, records that A has two outgoing links, and stores link relationships from A to B and from A to C. The graph can therefore preserve both the pages and the direction of their connections.
What a Restart Changes
The crawler is additive. Its database persists on disk, so a later run can use the records from an earlier run. If the first session collected 10 pages and a later session requests 20 pages, the crawler does not need to fetch the original 10 again. It skips pages already in the database and works toward the additional pages.
Tracing a Three-Page Crawl
Starting from example.com
Imagine an empty spider.sqlite database. You start spider.py at http://example.com/ and request three pages. Trace what is stored and how the crawler can continue.
- 1Start with the requested URL
The crawler begins with http://example.com/ as its starting point and fetches that page.
- 2Store the first page
The page receives a unique ID. Its URL and count of outgoing links are stored in the pages table. The links it contains are recorded as source-to-target relationships, and unvisited destinations become candidates for later crawling.
- 3Choose another unvisited link
The crawler selects an unvisited link discovered during the process and fetches that page. It stores the new page record and its outgoing link records in the same database.
- 4Continue until three pages are processed
The crawler repeats the same process for another unvisited page until the requested number of pages has been reached or no suitable unvisited page remains.
The database now contains page records and directed link records for the collected portion of the web. A later run can use this stored graph, skip pages already present, and continue from unvisited links.
Notice the distinction between discovery and storage. A link found in a page can be recorded as a relationship before its target page has been fetched and stored as a full page record. The crawler uses unvisited links as future work, gradually turning discovered connections into more complete page data.
Separate Starting Points
A single database can contain more than one starting point. For example, one session can begin at http://www.dr-chuck.com/ and another can begin at http://www.wikipedia.org/. These separate starting points are called webs within the program. Their pages and links share spider.sqlite even if the two collections are not directly connected.
The crawler does not maintain completely separate queues for separate webs. It treats all unvisited links as one unified queue and may interleave visits from the different starting points.
Common Mistakes
Common mistake Treating PageRank as the crawler itself.
What it looks likeAssuming that page importance can be calculated before pages and links have been collected.
Why it failsThe crawler creates the graph data first. Page rank is an analysis that can be run after the database contains pages and their connections.
Do this insteadSeparate the workflow into collection first and analysis second.
Common mistake Assuming that every discovered link has already been fetched.
What it looks likeSeeing a link record and concluding that the target page must already have a full page record.
Why it failsA discovered link can become an unvisited candidate before its target page is fetched.
Do this insteadDistinguish between recording a relationship and processing the target page.
Common mistake Expecting a second run to start from an empty database.
What it looks likeRunning the crawler again and interpreting skipped pages as a failure to crawl.
Why it failsThe crawler is designed to remember pages already stored and avoid re-crawling them.
Do this insteadTreat the existing database as accumulated crawl state. Delete spider.sqlite only when a fresh crawl is intended.
Common mistake Assuming different starting URLs require different database files.
What it looks likeCreating one database for each independent starting point.
Why it failsMultiple webs can coexist in the same
spider.sqlitedatabase.Do this insteadUse the shared database while remembering that the starting points may remain unconnected.
Common mistake Assuming the crawler always finishes one web before beginning another.
What it looks likeExpecting all pages reachable from the first starting URL to be processed before any page from the second starting URL.
Why it failsAll unvisited links are treated as one unified queue, so visits can be interleaved.
Do this insteadThink of the database as one growing graph with potentially separate clusters.
Practice the Data Flow
A database already contains pages collected from one starting URL. You run spider.py again with a different starting URL and request additional pages. Explain what happens to the existing page records, the new starting point, the unvisited queue, and the database structure.
Hints
- Ask whether stored pages are fetched again.
- Recall how the program treats multiple starting points.
- Identify what the pages table and links table gain during the later session.
What do you think happens?
If two starting URLs are not directly connected, can their pages still be stored in the same spider.sqlite database?
Reveal answer
Yes, as separate webs in one database
The crawler allows multiple independent starting points to coexist in the same database. It treats their unvisited links as one unified queue, even when the resulting webs are not directly connected.
Key Takeaways
- 1The crawler must collect pages and links before page-importance analysis can be performed.
- 2
spider.pyturns fetched pages into page records and directed link records inspider.sqlite. - 3The database persists between runs, allowing later sessions to skip pages already stored and add new pages incrementally.
- 4Separate starting URLs can form separate webs while sharing one database.
- 5The resulting page-and-link graph can later be queried, analyzed with page rank, or visualized with D3.
At a glance
- A crawler provides the data foundation for analyzing page importance.
- Each fetched page contributes both a page record and relationships to its outgoing links.
- Persistent storage makes crawling additive rather than redundant.
- Multiple independent starting points can share one database and one unified pool of unvisited links.
- PageRank analysis becomes meaningful only after this graph of pages and links has been collected.