Concepts / Socket Programming Basics

Socket Programming Basics

A socket does not necessarily deliver the complete HTTP response in one piece.

  • Programming
Interactive lab

Try it: Reading a Socket in Chunks

How the Python for Everybody socket client reads a web document: it sends a raw HTTP GET, then calls recv(n) in a loop, getting pieces of whatever has arrived, until an empty result says the server closed; the headers end at the first blank line (\r\n\r\n).

How it works

  1. Connect a socket to the server on port 80 and send 'GET http://host/path HTTP/1.0' followed by an empty line.
  2. Call recv(n): it returns at most n bytes — only what is waiting in the socket buffer — and blocks while nothing has arrived yet.
  3. Append each chunk to data; the chunk sizes depend on the buffer size and on packet timing.
  4. When recv returns b'' the server has closed the connection, so break out of the loop.
  5. Find the first b'\r\n\r\n': before it are the status line and headers, after it is the document (Content-Length bytes).

Default run (19 steps): The client will fetch http://data.pr4e.org/romeo.txt with a raw socket and read the reply 64 bytes at a time. The server's reply is 335 bytes and will arrive in 4 packets of up to 100 bytes. … Split: data[:164] is the status line and headers; data[164+4:] is the 167-byte body — the document itself (its Content-Length says 167).

Simplified: The network is a model on a simulated clock: the reply arrives in equal-size packets at a fixed interval and the server closes with the last packet. The response headers are a fixed template. Nothing is sent over a real network.

Educational simulation

Loading the simulation…

From Socket to Saved Image

Retrieving an image over HTTP is a movement of bytes through several stages. A program connects to a server, sends an HTTP GET request, receives the response in chunks, combines those chunks into one bytes buffer, removes the HTTP headers, and writes the remaining binary body to a file. The essential discipline is to keep the response as bytes from the moment it arrives until it is written to disk.

HTTP socketincoming response bytesbytes bufferheaders plus image bodybinary bodyimage bytesoutput filebinary write modereceive chunksskip headerswrite bytes
How does image data move from the HTTP socket, through the program's buffer, and into the output file?

The Three-Stage Retrieval Pattern

  1. 1
    Establish a socket connection and send an HTTP GET request for the binary file.
  2. 2
    Receive response chunks and concatenate them into an initially empty bytes buffer until the server closes the connection.
  3. 3
    Find the double CRLF boundary, extract the bytes after it, and write those bytes to a file opened in binary write mode.
Client programHTTP serverSocketOutput fileHTTP GET requestresponse chunksreceived bytesbinary body
What sequence does the program follow from connecting to the server and sending the request to saving the response body?

Accumulating Response Chunks

A socket does not necessarily deliver the complete HTTP response in one piece. The program receives data in chunks, typically 5120 bytes at a time, and adds each chunk to a growing buffer. The buffer begins as an empty bytes object, represented by b"". Each received chunk is concatenated to the existing buffer. This continues until the receive operation returns fewer than 1 byte, indicating that the server has closed the connection and no more data is available.

Empty bufferb""Chunk 1received bytesPartial responsebuffer plus chunk 1Chunk 2more received bytesComplete responseheaders plus bodyconcatenateaddconcatenate more chunksadd
What happens to each incoming chunk of bytes, and how do the chunks combine into one complete response?

Finding the Header Boundary

After the socket closes, the buffer contains two different parts: the HTTP response headers and the binary image body. HTTP separates these parts with a blank line represented by two consecutive carriage return and line feed sequences: \r\n\r\n. Search the bytes buffer for this exact marker with find(b'\r\n\r\n'). The find operation returns the index where the marker begins. Because the marker occupies 4 bytes, the body begins at position pos + 4. Slicing from that position onward discards the headers and keeps only the image bytes.

HTTP headersmetadata\r\n\r\n4-byte boundaryBinary bodyimage bytesends beforeskip 4 bytes
Where does the double CRLF marker occur, and how does it divide the HTTP headers from the image bytes?
Worked example

Separating One HTTP Response

A complete response buffer contains HTTP headers, the marker \r\n\r\n, and an image body. How should the program identify the bytes to save?

  1. 1
    Locate the marker

    Search the bytes buffer with find(b'\r\n\r\n'). The returned position is the start of the four-byte separator.

  2. 2
    Move past the separator

    Add 4 to the returned position because the double CRLF marker is four bytes long.

  3. 3
    Extract the body

    Slice the buffer from position + 4 onward. The resulting bytes contain the binary image body without the HTTP headers.

Result

The saved data must be the response buffer sliced from the first byte after the double CRLF marker through the end of the buffer.

Writing Bytes Without Corruption

Once the body has been extracted, open the destination file in binary write mode, written as wb, and write the binary data to it. Binary write mode avoids encoding transformations that can corrupt the received data. The HTTP headers should not be written to the file; only the bytes after the double CRLF marker belong in the image file.

wbbinary write modeText modeencoding transformationsImage bytespreserved dataCorrupted dataunreliable image filewrite unchanged bytesmay transform data
What is the difference between writing received bytes in binary mode and treating them as text?
Best practice

Check the Content-Type header before saving the body. Values such as image/jpeg or image/png indicate an image response. If the server instead returns text/html, the request may have gone wrong, and saving the response as an image would be inappropriate.

Common Retrieval Mistakes

  • Common mistake

    Initializing the response buffer as text instead of bytes.

    What it looks like

    Using an empty string for data that will contain an image response.

    Why it fails

    Binary response data must be accumulated in a bytes buffer, not handled as text.

    Do this instead

    Start with an empty bytes object, represented by b"".

  • Common mistake

    Saving the complete response buffer.

    What it looks like

    Writing the headers and the image body together to the output file.

    Why it fails

    The file begins with HTTP metadata rather than only the binary image body.

    Do this instead

    Find the double CRLF marker and write only the slice beginning at position + 4.

  • Common mistake

    Skipping the wrong number of bytes after the marker.

    What it looks like

    Starting the body slice at pos or pos + 2.

    Why it fails

    The double CRLF marker is four bytes long.

    Do this instead

    Begin the body slice at pos + 4.

  • Common mistake

    Hard-coding the header length.

    What it looks like

    Always removing a fixed number of bytes before saving.

    Why it fails

    The header-body boundary should be located by searching for the marker.

    Do this instead
    Use find(b'\r\n\r\n') to determine the boundary.
  • Common mistake

    Opening the destination in text mode.

    What it looks like

    Writing image data with a text-oriented file mode.

    Why it fails

    Encoding transformations can corrupt binary data.

    Do this instead

    Use binary write mode, wb.

Trace the Complete Process

MEDIUM

Describe the path of an image response in your own words. Include what the initial buffer contains, what happens to each received chunk, how the header-body boundary is found, which bytes are discarded, and which file mode is used for the final write.

Hints
  • The response buffer eventually contains both headers and body.
  • The boundary marker is the byte pattern \r\n\r\n.
  • The marker is four bytes long, so the body begins at position + 4.
  • The destination must be opened in binary write mode.
StageData heldAction
ReceiveIncoming response chunksConcatenate each chunk to the bytes buffer
SeparateHeaders followed by binary bodyFind the double CRLF marker and skip four bytes
SaveBinary body onlyWrite it to a file in wb mode
VerifyHTTP metadataCheck Content-Type before saving

The state of the data at each major stage

Key Takeaways

  1. 1
    Accumulate socket data in a bytes buffer initialized as b"".
  2. 2
    Continue receiving and concatenating chunks until the server closes the connection.
  3. 3
    Find the HTTP header-body boundary with find(b'\r\n\r\n').
  4. 4
    Because the marker is four bytes long, extract the body from position + 4 onward.
  5. 5
    Write only the binary body to a file opened in binary write mode, wb, and check Content-Type when verifying the response.

At a glance

  • A complete HTTP response contains headers followed by a binary body.
  • Socket data arrives in chunks, so the chunks must be concatenated into a growing bytes buffer.
  • The double CRLF marker, \r\n\r\n, separates the headers from the body.
  • The binary body begins four bytes after the marker and should be written using binary write mode, wb.
  • Checking Content-Type helps confirm that the response is the expected image rather than an error page.

Related Topics

Related concepts

More concepts starting with S