Building Network Applications: From HTTP to Web Scraping
Network communication therefore requires more than a connection: the programs need agreed rules about what happens next.
The Waiting Problem
Imagine two programs that need to exchange information. One program tries to read, but the other program has not sent anything yet. The reading program cannot obtain data that does not exist, so it waits. If the second program is also waiting for the first program to send something, neither program makes progress. Network communication therefore requires more than a connection: the programs need agreed rules about what happens next.
What do you think happens?
Program A reads from a socket before Program B has sent any data. What is the most likely result?
Reveal answer
Program A blocks and waits
When no data is available on the socket, reading causes the program to wait for the other program to send data.
Sockets and Local Files
A socket is a two-way connection between two programs. Both programs can read data from the connection and write data to it. This makes a socket similar to a file in the sense that a program can use reading and writing operations, but the destination is different: a file connects a program with local stored data, while a socket connects a program with another remote program.
| Aspect | Local file | Socket |
|---|---|---|
| What it connects to | Local stored data | A remote program |
| Reading | Reads locally stored data | Reads data exchanged by programs |
| Writing | Writes local stored data | Writes data for the other program to receive |
| Direction of interaction | Uses file reading and writing | Allows both programs to read and write |
A socket resembles a file interface, but it connects programs rather than providing local storage.
The Communication Sequence
A Request-and-Response Exchange
Trace one orderly exchange between Program A and Program B over a socket.
- 11. Establish the connection
The two programs use a socket as their two-way connection.
- 22. Follow the agreed first action
The protocol determines which program sends first and what that first message should contain.
- 33. Receive the message
The other program reads the data that was sent rather than reading before any data is available.
- 44. Send the expected response
The receiving program follows the protocol and sends the response that the first program expects.
- 55. Continue in the defined order
Each program knows whether it should send, receive, or wait for the next step.
The programs exchange data because the connection and the communication order work together.
The important sequence is not simply send everything at once. The programs must first have a connection, then use the protocol to determine who sends first, what the message means, what response is expected, and who acts next. A successful exchange is therefore coordinated activity between both programs.
Protocols as Shared Rules
A protocol is a set of precise rules that determines the sequence of communication: who sends first, what they send, what response is expected, and who sends next.
A socket provides the two-way connection, but it does not by itself tell the programs how to coordinate. The protocol supplies that coordination. When both programs follow the same sequence, a program reads when the protocol says data should be available and sends when the protocol says it is its turn. This prevents the situation in which both programs wait for the other to act.
Two-Way Data Flow
A socket supports movement in both directions. Program A can write data that Program B reads, and Program B can write data that Program A reads. The same connection therefore supports an exchange rather than a one-direction-only transfer. Each direction still depends on the protocol: a program should know whether it is currently expected to send or receive.
In an exchange such as a request followed by a response, one program writes the request through the socket, the other program reads it and writes a response, and the first program reads that response. The socket makes both directions possible; the protocol determines the order.
Mistakes Beginners Make
Common mistake Treating a socket as if it were only a local file.
What it looks likeAssuming that reading from a socket simply retrieves already stored local data.
Why it failsA socket connects a program to another remote program, and the other program may not have sent data yet.
Do this insteadRemember that socket data is exchanged between programs and that reading can wait for the other program.
Common mistake Assuming that a read always returns immediately.
What it looks likeProgram A reads before Program B sends anything.
Why it failsWhen no data is available, the reading program blocks and waits.
Do this insteadTrace the protocol first and identify when the other program is expected to send.
Common mistake Allowing both programs to wait for the other to send first.
What it looks likeProgram A waits for a message from Program B while Program B waits for a message from Program A.
Why it failsBoth programs can remain waiting, creating a deadlock.
Do this insteadUse a protocol that explicitly specifies who sends first and what happens next.
Common mistake Thinking that the connection alone defines the conversation.
What it looks likeConnecting two programs without agreeing on message order.
Why it failsThe socket provides the connection, but the protocol provides the precise communication sequence.
Do this insteadSeparate the role of the socket from the role of the protocol.
When analyzing a socket interaction, write down the next action for each program: send, receive, or wait. Then check whether the protocol guarantees that the expected data will be sent before a read occurs.
Practice the Trace
Program A and Program B have a socket connection. Their protocol says that Program A sends a request first, Program B reads the request and sends a response, and Program A reads the response. Describe the action sequence and identify why neither program should wait indefinitely.
Hints
- Start with the program named as the first sender.
- After each send, identify which program is expected to read.
- Look for the point at which the protocol gives each program a defined next action.
Practice Solution
Explain the request-and-response sequence and why it avoids indefinite waiting.
- 1First
Program A sends the request because the protocol identifies Program A as the first sender.
- 2Second
Program B reads the request because the request has been sent and the protocol says Program B should receive it.
- 3Third
Program B sends the response required by the protocol.
- 4Fourth
Program A reads the response because Program B has sent it.
Neither program waits indefinitely because the protocol specifies who sends first, what response follows, and which program reads next.
Key Takeaways
- 1A socket is a two-way connection between two programs.
- 2A socket resembles a file interface, but it exchanges data with a remote program instead of accessing local stored data.
- 3Reading from a socket when no data is available causes the program to block and wait.
- 4If both programs wait for the other to send, communication can reach a deadlock.
- 5A protocol prevents this problem by defining who sends first, what is sent, what response is expected, and who acts next.
At a glance
- Sockets connect programs for two-way reading and writing, while files provide access to local stored data.
- A socket read can block when the other program has not sent data.
- Two programs can deadlock if both wait for the other to send first.
- Protocols coordinate communication by defining the order of sending, receiving, responding, and waiting.
- A successful exchange follows a clear sequence from connection through message exchange.