Concepts / Web Crawling with spider.py

Web Crawling with spider.py

If you run spider.py again, the spider.json database can grow or change as new pages are discovered or link information is updated.

  • Programming

From Crawl to Network

A web crawl produces data, but the data becomes easier to explore when it is turned into a visual network. In this workflow, spider.py discovers pages or updates link information in the PageRank database stored in spider.json. Next, spjson.py extracts information about the most highly linked pages and writes JSON data. Finally, force.html loads that JSON so D3.js can display an interactive graph.

spider.pycrawl or update pagesspider.jsonPageRank databasespjson.pywrite JSONforce.htmlD3.js graphupdatesreadsproduces data for
What happens next after a crawl, and in what order are spider.py, the database, spjson.py, and force.html used?

The Conversion Step

spjson.py is the bridge between stored crawl results and the visualization. When you execute it from a command line or terminal, it reads the PageRank database in spider.json. It extracts information about the most highly linked pages and writes that information in JSON format. D3.js can then read this JSON and render the network graph.

PageRank databasespider.jsonHighly linked pagesextracted informationNetwork JSONnodes and linksD3.jsreads JSONspjson.py readswritesis read by
How does data move from the PageRank database through spjson.py into the JSON consumed by force.html?
Best practice

Treat spjson.py as a regeneration step rather than as a manual editing task. The source material says that you do not need to edit the JSON file manually. Instead, let spjson.py create the data that D3.js consumes.

Nodes and Links

The generated JSON has two main arrays: nodes and links. A node represents a web page and includes properties such as its URL and PageRank score. A link represents a connection between two pages and is typically specified using the indices or IDs of the source and target nodes. D3.js uses these two parts together: it positions the nodes and draws lines for the links.

nodesweb pagesPage AURL and PageRank scorelinkspage connectionsPage BURL and PageRank scorePage A to Page Bsource and targetcontainscontainscontainssourcetarget
How are highly linked pages represented as nodes and links in the network JSON?
Worked example

Following One Connection

Suppose the generated JSON contains a node for Page A, a node for Page B, and a link whose source is Page A and whose target is Page B. What does that part of the graph represent?

  1. 1
    Read the nodes

    The two nodes represent two web pages. Each node can include the page URL and its PageRank score.

  2. 2
    Read the link

    The link identifies Page A as the source and Page B as the target, using the nodes' indices or IDs.

  3. 3
    Read the rendered graph

    D3.js uses the node records and the link record to position the pages and draw a connection between them.

Result

The visual connection represents the relationship encoded by the link between the two page nodes.

The Force-Directed View

After force.html opens in a browser, the D3.js visualization is live and interactive. The graph uses a force-directed layout. Forces push nodes apart to help avoid overlap, while links pull connected nodes together. The result is an automatic arrangement intended to make the network readable rather than a fixed list of pages.

Automatic layoutnodes and links positionedSelected nodenew positionPage connectionslinks remain visibleclick and dragshows relationships
How do dragging nodes, zooming, and changing the layout affect the positions and connections in the graph?
Key point

Moving a node changes where it appears in the view, but the reason for moving it is interpretive: you can untangle overlapping nodes or focus on a particular region of the graph. The links continue to show the encoded connections between pages.

Reading a Node

A visible circle or node is not enough to tell you which real web page it represents. Double-click a node to reveal its URL. That interaction connects the abstract graph element to the actual page identity. After identifying it, click and drag the node if you need to separate it from nearby nodes or examine its connected region.

Selected nodeweb pageURLrevealed by double-clickConnected page Alinked nodeConnected page Blinked nodedouble-click revealslinklink
Which URL does a selected node represent, and how is that node connected to other pages?

What do you think happens?

You see a node in force.html but do not know which page it represents. Which interaction identifies the page?

  • Click and drag the node

  • Double-click the node

  • Run spider.py without refreshing

  • Edit the JSON manually

Reveal answer
Answer:

Double-click the node

The visualization reveals the URL represented by a node when you double-click it. Dragging changes the node's position instead.

Refreshing New Crawl Data

The visualization reflects the data that spjson.py has generated. If you run spider.py again, the spider.json database can grow or change as new pages are discovered or link information is updated. The browser will not reflect those changes merely because the database changed. Run spjson.py again to regenerate the JSON, then press refresh in the browser to reload the visualization.

spider.jsonearlier crawl dataNetwork graphearlier JSONspider.jsonupdated crawl dataNetwork JSONregenerated by spjson.pyNetwork graphbrowser refreshedearlier dataspjson.py convertsrefresh loads
What changes in the graph when new crawl results are converted into JSON and loaded into force.html?

Common Workflow Mistakes

  • Common mistake

    Opening force.html before generating the JSON data

    What it looks like

    The PageRank database exists, but spjson.py has not been executed.

    Why it fails

    The visualization depends on the JSON output that spjson.py writes for D3.js to read.

    Do this instead

    Execute spjson.py first, then open force.html in the browser.

  • Common mistake

    Assuming a node's position identifies its URL

    What it looks like

    A learner sees a prominent node and guesses which web page it represents.

    Why it fails

    The graph position does not by itself reveal the real page identity.

    Do this instead

    Double-click the node to reveal its URL.

  • Common mistake

    Refreshing the browser without regenerating the JSON

    What it looks like

    spider.py has discovered new pages, but the learner only presses refresh in force.html.

    Why it fails

    The visualization still has the previously generated JSON data.

    Do this instead

    Run spjson.py again and then refresh the browser.

  • Common mistake

    Treating links as separate pages

    What it looks like

    A learner counts a line as though it were another node.

    Why it fails

    Nodes represent web pages, while links represent connections between pages.

    Do this instead

    Read nodes as pages and links as relationships between source and target nodes.

Practice the Pipeline

MEDIUM

A new crawl has been completed with spider.py. Describe the exact sequence you would use to make the updated network visible in the browser. Then explain what you would do if you wanted to identify one node and separate it from overlapping nodes.

Hints
  • Start with the PageRank database and the script that converts it to JSON.
  • Remember that the browser needs a refresh after the JSON is regenerated.
  • Use double-clicking for URL identification and click-and-drag for repositioning.
Worked example

Checking an Updated Graph

You have run spider.py again and want force.html to show the updated crawl.

  1. 1
    Update the stored data

    The new crawl changes or grows the PageRank database stored in spider.json.

  2. 2
    Regenerate the visualization data

    Execute spjson.py so it reads the updated database and writes new JSON containing nodes and links for the highly linked pages.

  3. 3
    Reload the graph

    Press refresh in the browser while force.html is open so the visualization loads the regenerated data.

  4. 4
    Inspect a page

    Double-click a node to reveal its URL, or click and drag it to reposition the node for easier inspection.

Result

The browser shows the regenerated network, and node interactions help you identify pages and examine their connections.

Key Takeaways

  1. 1
    spider.py discovers pages or updates link information in the PageRank database stored in spider.json.
  2. 2
    spjson.py reads that database and writes JSON containing nodes for web pages and links for connections between them.
  3. 3
    force.html loads the JSON so D3.js can display a force-directed network graph.
  4. 4
    Double-click a node to reveal its URL, and click and drag a node to reposition it.
  5. 5
    After a new crawl, run spjson.py again and refresh the browser to display the updated network.

At a glance

  • The workflow connects spider.py, spider.json, spjson.py, and force.html in that order.
  • spjson.py converts PageRank database information about highly linked pages into JSON for D3.js.
  • The JSON separates page nodes from the links that connect them.
  • The force-directed graph can be explored by dragging nodes and double-clicking them to reveal URLs.
  • Updated crawl results require both regenerating the JSON and refreshing the browser.

More concepts starting with W