Concepts / Understanding PageRank Calculations

Understanding PageRank Calculations

PageRank depends on page relationships, not only on a raw link count.

  • Programming

From Crawl Data to Readable Evidence

After a web spider finishes crawling, it stores its collected data in spider.sqlite. That file is a binary SQLite database, so it is not a convenient human-readable view of the pages and connections the spider found. The spdump.py tool reads spider.sqlite and prints a summary that lets you inspect page connectivity and PageRank-related values.

The most useful way to approach spdump.py is as a validation window into the crawl. It does not merely list URLs. Each output line describes one page using five values: incoming_links, old_rank, new_rank, page_id, and url. Reading those values in the correct order helps you check whether the crawler discovered the relationships you expected.

Reading the Five Positions

Every line follows the same structure: (incoming_links, old_rank, new_rank, page_id, url). Position matters. The first value is not the page ID, and the final value is not a rank. Treat the tuple as an ordered record: each position answers a different question about the page.

incoming_linksposition 1old_rankposition 2new_rankposition 3page_idposition 4urlposition 5
Which position in the five-element tuple contains each piece of page information?
PositionFieldMeaning
1incoming_linksThe number of other pages in the crawl that link to this page.
2old_rankThe earlier rank value; it is often None on the first run.
3new_rankThe newer PageRank value shown by the tool.
4page_idThe page's identifier in the stored crawl data.
5urlThe page address.

The five fields printed for each page

Tracing One Tuple

Worked example

Interpreting a sample output line

Interpret the tuple (5, None, 1.0, 3, url).

  1. 1
    First position

    The value 5 is incoming_links. Five distinct pages in the crawled network contain a hyperlink pointing to this page.

  2. 2
    Second position

    The value None is old_rank. The source describes this as common on the first run, before numeric rank values have appeared after multiple PageRank computations.

  3. 3
    Third position

    The value 1.0 is new_rank, the newer rank value displayed for the page.

  4. 4
    Fourth position

    The value 3 is page_id, the identifier associated with this page in the crawl data.

  5. 5
    Fifth position

    The value url identifies the page address represented by the tuple.

Result

This tuple describes a page with five incoming links, no earlier numeric rank recorded in the displayed state, a new rank of 1.0, page ID 3, and the displayed URL.

Key point

The first value is the direct connectivity measure. It tells you how many other pages in the crawl link to the page represented by the rest of the tuple.

Why Some Pages Disappear

spdump.py does not print every page the spider discovered. Its output includes only pages with at least one incoming link. A page with zero inbound connections is filtered out by design, so its absence from the output does not by itself mean that the spider failed to discover it.

Pages inspider.sqliteIncoming linksat least one?spdump.py outputpage is shownZero inbound linkspage is filtered outreadyesno
What happens between pages stored in spider.sqlite and pages printed by spdump.py?

The filtering reflects a connectivity distinction. Some discovered pages may have outgoing links but no other pages in the crawled network linking to them. These are leaf or terminal pages in the network. Because they have zero incoming links, they cannot receive PageRank from other pages through inbound connections, so spdump.py keeps its displayed network focused on pages that participate in those connections.

Edge case

If you expect a page to appear but cannot find it in spdump.py, first consider whether the page has any inbound links in the crawled network. Its absence may be the result of the output filter rather than a crawling failure.

Using Connectivity to Validate a Crawl

Use the first tuple value to compare the observed structure with your expectations. A page with many incoming links is highly connected within the crawled network. Pages with few incoming links are less connected. This information can reveal whether the spider seems to have captured the relationships you expected from the site.

  • Check whether an expected central page has an incoming-link count that matches your expectations.
  • If a page has fewer incoming links than expected, consider whether the spider crawled deeply enough or missed pages.
  • Use page_id and url together to identify which stored page a tuple describes.
  • Compare old_rank and new_rank to observe whether rank values have progressed beyond the initial state.
  • Remember that a page absent from output may have zero inbound connections and therefore may have been filtered out.
Unexpected observationWhat to inspect
A page has fewer incoming links than expectedCheck whether the spider crawled deeply enough or missed pages.
An expected page is absentConsider whether it has zero inbound connections and was filtered out.
old_rank is NoneConsider whether this is the first run before multiple PageRank computations.
A URL or identifier looks unfamiliarUse the url and page_id fields to trace the displayed record back to the crawl data.

Practical checks when validating spdump.py output

Common Reading Mistakes

  • Common mistake

    Treating the tuple as an unordered collection of values.

    What it looks like

    Reading the first value as page_id because it is the first number shown.

    Why it fails

    Each position has a fixed meaning. The first position is incoming_links, while page_id is the fourth position.

    Do this instead

    Always map the tuple using the order incoming_links, old_rank, new_rank, page_id, url.

  • Common mistake

    Assuming an absent URL was never discovered.

    What it looks like

    Concluding that a page is missing from the crawl because it does not appear in spdump.py output.

    Why it fails

    spdump.py filters out pages with zero incoming links.

    Do this instead

    Check whether the page may have zero inbound connections before diagnosing a crawl failure.

  • Common mistake

    Treating None in old_rank as a corrupted rank.

    What it looks like

    Reporting the first-run value None as an invalid numeric result.

    Why it fails

    old_rank is often None on the first run; numeric values appear after PageRank has been computed multiple times.

    Do this instead

    Interpret None in the context of the calculation stage.

  • Common mistake

    Using incoming_links as the complete PageRank explanation.

    What it looks like

    Assuming the page with the most incoming links must have the highest rank.

    Why it fails

    The exact PageRank calculation also depends on the PageRank of the pages linking to the page.

    Do this instead

    Use incoming_links as a connectivity measure and inspect old_rank and new_rank as the rank fields.

Practice the Tuple Trace

EASY

Suppose an spdump.py line has the form (2, None, 1.0, 7, url). Without looking back at the table, identify what each of the five positions tells you. Then explain two reasons why a different crawled page might not appear in spdump.py output.

Hints
  • Use the fixed field order rather than the data type of each value.
  • Remember the output filter for pages with zero incoming links.
  • Remember the special first-run meaning of old_rank being None.
Worked example

Practice answer

Interpret (2, None, 1.0, 7, url) and explain why another page might be absent.

  1. 1
    Connectivity

    The first value, 2, means two other pages in the crawled network link to this page.

  2. 2
    Earlier rank

    The second value, None, is old_rank and is often seen on the first run.

  3. 3
    Newer rank

    The third value, 1.0, is new_rank.

  4. 4
    Identity

    The fourth value, 7, is page_id, and the fifth value is the page URL.

  5. 5
    Filtering

    Another crawled page may be absent because it has zero incoming links and spdump.py deliberately filters it out.

Result

The tuple describes a page with two incoming links, an old_rank of None, a new_rank of 1.0, page ID 7, and the displayed URL. A page with no inbound connections would not be printed.

What to Remember

  1. 1
    spdump.py reads spider.sqlite and presents selected crawl data in a human-readable form.
  2. 2
    Every displayed record follows the order incoming_links, old_rank, new_rank, page_id, url.
  3. 3
    incoming_links directly measures how many other crawled pages link to the page.
  4. 4
    Only pages with at least one incoming link appear; pages with zero inbound connections are filtered out.
  5. 5
    old_rank is often None on the first run, while later numeric values show the progression of PageRank calculations.
  6. 6
    Use the output to validate page connectivity, investigate unexpected URLs or counts, and check whether the crawl matches your expectations.

At a glance

  • spdump.py provides a readable view of connectivity and rank-related data stored in spider.sqlite.
  • The five tuple positions always mean incoming_links, old_rank, new_rank, page_id, and url.
  • The incoming-link count shows how many other pages in the crawl link to the displayed page.
  • Pages with zero incoming links are intentionally excluded from the output.
  • Comparing the tuple fields with your crawl expectations helps debug and validate the spider's results.

Related Topics

Related concepts

More concepts starting with U