Concepts / Analyzing Network Connectivity Patterns

Analyzing Network Connectivity Patterns

Visualizing a network graph requires running spjson.py to convert your PageRank database into JSON data, then opening force.html in a browser to view the D3.js visualization.

  • Programming

From Crawl Data to a Graph

A PageRank database contains information about pages discovered during a web crawl and the links between those pages. That information is useful for analysis, but its relationships are easier to explore visually. The visualization workflow has two main transformations: spjson.py reads the PageRank database stored in spider.json and writes JSON data, then force.html loads that data so D3.js can render an interactive network graph.

spider.jsonPageRank databasespjson.pyextracts highly linkedpagesJSON datanodes and linksforce.htmlD3.js visualizationNetwork graphinteractive nodes and linksreadswritesis read byrenders
How does crawl data move from the PageRank database to the nodes and links displayed in the graph?

The JSON Graph Model

The generated JSON describes the network with two main arrays: nodes and links. A node represents a web page. Its information includes properties such as the page URL and its PageRank score. A link represents a connection between two pages, usually by referring to the indices or IDs of the source and target nodes. D3.js consumes these arrays to create the visible points and lines of the graph.

Worked example

Reading a Small Network

Suppose the generated data contains three page nodes and links connecting the first page to the second and the second page to the third. What would the visualization represent?

  1. 1
    Identify the nodes

    Each of the three node entries represents one web page. The URL identifies the real page, while the PageRank score provides information associated with that page.

  2. 2
    Identify the links

    Each link entry identifies a source page and a target page. Those relationships become connections drawn between the corresponding nodes.

  3. 3
    Interpret the graph

    The result is a connected chain of three page nodes. The graph does not merely list pages; it displays how the pages are connected.

Result

The nodes represent pages and the links represent relationships between those pages. D3.js uses both parts together to render the network.

Page Ahighly linked pagePage Bconnected pagePage Cconnected pagePage Dless connected pagelinklinklink
Which pages are connected, and how can a highly linked page differ from a page with fewer connections?

Running the Visualization Workflow

  1. 1
    Run spjson.py from the command line or terminal.
  2. 2
    Let the script read the PageRank database stored in spider.json.
  3. 3
    Use the JSON output containing nodes for pages and links for connections.
  4. 4
    Open force.html in a browser.
  5. 5
    Inspect the live D3.js force-directed visualization.

The workflow separates data preparation from display. spjson.py prepares a format that D3.js can read; force.html provides the browser-based visualization. When force.html opens, the graph is arranged automatically by a force-directed algorithm. Forces push nodes apart to reduce overlap, while links pull connected nodes together. The resulting arrangement is intended to make the network easier to read, not to represent a fixed geographic layout.

PageRank databasespider.jsonData extractionspjson.pyBrowser displayforce.htmlInteractive graphD3.js networkinputgenerated datarenders
What happens as you move from the database to the interactive graph?

Exploring Nodes and Layout

Once the graph is visible, begin with the structure rather than trying to identify every page immediately. Look for nodes with many visible links and compare them with nodes that have fewer connections. Then use interaction to investigate. Click and drag a node to reposition it. This can untangle overlapping regions or bring a particular part of the graph into focus. Double-click a node when you need to discover the URL represented by that node.

Worked example

Investigating an Unclear Cluster

A group of nodes overlaps in the force-directed graph, and one node appears to have several connections. How can you investigate it?

  1. 1
    Reposition the node

    Click and drag the node away from the overlap. Moving it can make its connecting links easier to see.

  2. 2
    Inspect its identity

    Double-click the node to reveal its URL. This identifies the real web page represented by the node.

  3. 3
    Compare its connections

    Use the separated layout to examine which other page nodes are connected to it. The visible links show the node’s relationships in the generated network.

Result

Dragging improves visibility, while double-clicking connects the visual node to the real page URL.

Page Aoverlapping regionPage ArepositionedPage Boverlapping regionPage Bseparated viewexisting linksame connection
What changes when a learner drags a node, and what remains the same about the network?

Keeping the Graph Current

The visualization reflects the data that was generated for it. If spider.py is run again to discover new pages or update link information, spider.json grows or changes. The existing graph does not automatically represent those updates. Run spjson.py again so the updated database is converted into new JSON data, then press refresh in the browser while viewing force.html.

Before

Earlier crawl data is displayed in force.html.

The graph reflects the earlier spider.json data.

After

Run spjson.py again, then refresh force.html in the browser.

The graph reflects the regenerated JSON data from the updated crawl.

Mistakes to Avoid

  • Common mistake

    Opening force.html without regenerating data after a new crawl

    What it looks like

    A new crawl changes spider.json, but the browser still shows the earlier network.

    Why it fails

    The visualization uses the JSON data produced for it; updated database contents must first be converted again.

    Do this instead

    Run spjson.py again and then refresh the browser.

  • Common mistake

    Treating a node’s position as its identity

    What it looks like

    Assuming that a node near the center must represent a particular page.

    Why it fails

    The force-directed layout arranges nodes to make connections readable, and position alone does not reveal the represented URL.

    Do this instead

    Double-click the node to reveal its URL.

  • Common mistake

    Confusing nodes with links

    What it looks like

    Counting a line as if it were another page.

    Why it fails

    Nodes represent web pages, while links represent connections between pages.

    Do this instead

    Read the graph as page nodes connected by relationship links.

  • Common mistake

    Leaving overlapping nodes untouched

    What it looks like

    Trying to inspect several links that are hidden in a crowded region.

    Why it fails

    Overlap can make the network structure difficult to read.

    Do this instead

    Click and drag nodes to reposition them and untangle the view.

Apply the Workflow

EASY

Describe the steps you would take after spider.py has discovered additional pages and updated link information. Include the data file, the script, the browser page, and the final interaction needed to verify a node’s identity.

Hints
  • Start with the updated PageRank database stored in spider.json.
  • Remember that spjson.py must be run again before the browser is refreshed.
  • Use double-clicking when you need to reveal a node’s URL.
  1. 1
    A network visualization begins with PageRank data in spider.json. Running spjson.py extracts information about the most highly linked pages and writes JSON containing nodes and links. Opening force.html lets D3.js display that data as a force-directed graph. Drag nodes to improve the layout, double-click them to reveal their URLs, and regenerate the JSON plus refresh the browser whenever a new crawl changes the underlying data.

Key Takeaways

  • spjson.py converts PageRank database information from spider.json into JSON data for visualization.
  • Nodes represent web pages, and links represent connections between those pages.
  • force.html displays the data as an interactive D3.js force-directed graph.
  • Dragging changes the visible arrangement, while double-clicking reveals a node’s URL.
  • After a new crawl, run spjson.py again and refresh the browser to display updated data.

Related Topics

Related concepts

More concepts starting with A