Concepts / Handling Binary Data and Encoding in Python

Handling Binary Data and Encoding in Python

A Python string is text, but a socket works with bytes.

  • Programming
Interactive lab

Try it: Reading a Socket in Chunks

How the Python for Everybody socket client reads a web document: it sends a raw HTTP GET, then calls recv(n) in a loop, getting pieces of whatever has arrived, until an empty result says the server closed; the headers end at the first blank line (\r\n\r\n).

How it works

  1. Connect a socket to the server on port 80 and send 'GET http://host/path HTTP/1.0' followed by an empty line.
  2. Call recv(n): it returns at most n bytes — only what is waiting in the socket buffer — and blocks while nothing has arrived yet.
  3. Append each chunk to data; the chunk sizes depend on the buffer size and on packet timing.
  4. When recv returns b'' the server has closed the connection, so break out of the loop.
  5. Find the first b'\r\n\r\n': before it are the status line and headers, after it is the document (Content-Length bytes).

Default run (19 steps): The client will fetch http://data.pr4e.org/romeo.txt with a raw socket and read the reply 64 bytes at a time. The server's reply is 335 bytes and will arrive in 4 packets of up to 100 bytes. … Split: data[:164] is the status line and headers; data[164+4:] is the 167-byte body — the document itself (its Content-Length says 167).

Simplified: The network is a model on a simulated clock: the reply arrives in equal-size packets at a fixed interval and the server closes with the last packet. The response headers are a fixed template. Nothing is sent over a real network.

Educational simulation

Loading the simulation…

From Request to Response

A web browser can be understood as a sequence of low-level operations: create a socket, connect to a web server, send an HTTP request, receive the response in pieces, decode those pieces, and close the connection. This article follows that sequence so you can see how binary data moves through a socket and becomes readable text.

Key point

The socket carries bytes. The HTTP request is written as text first, encoded into bytes before sending, and received bytes are decoded into text before printing.

Opening the Connection

The program begins by creating a socket object and connecting it to a server on port 80. Port 80 is the standard port for HTTP, which is the unencrypted form of web traffic described in this exercise. Calling connect() starts a TCP three-way handshake: the client sends SYN, the server replies with SYN-ACK, and the client sends ACK. After this exchange, the socket has a reliable communication channel to the server's listening socket.

ClientWeb serverPort 80SYNSYN-ACKACK
How does the client establish a reliable channel before sending the HTTP request?
python

Formatting the HTTP Request

Once the socket is connected, the client sends an HTTP GET request. The request starts with GET, includes a resource path and an HTTP version, and ends with two consecutive line endings. The HTTP protocol interprets the blank line represented by \r\n\r\n as the end of the request headers.

GETrequest method/resource pathHTTP versionprotocol version\r\nline ending\r\nblank linethenthenthenthen
What lines and end-of-line sequences make up the request sent to the server?

The sequence \r\n is a carriage return followed by a line feed. The carriage return is ASCII 13 and the line feed is ASCII 10. In \r\n\r\n, the first pair ends the request line and the second pair creates a blank line. That blank line tells the server that no more headers are coming and that the request is complete.

python
Key point

A Python string is text, but a socket works with bytes. Calling encode() converts the request string into bytes before it is sent.

Reading the Response Stream

The response does not necessarily arrive as one complete object. It arrives as a stream of packets, so the client repeatedly calls recv(512). Each call retrieves up to 512 bytes. The program processes each returned chunk and continues calling recv() until the server has finished sending and the call returns an empty result.

Server responsebyte streamChunk 1up to 512 bytesChunk 2up to 512 bytesChunk nup to 512 bytesDecoded outputprocessed in orderEmpty resultstop receivingrecv(512)decode and printreceive againdecode and printrepeatserver closes
How does the client repeatedly receive partial byte chunks and know when the server has finished?
Code walkthrough
while True:
    chunk = sock.recv(512)
    if not chunk:
        break
    print(chunk.decode(), end='')
  1. 1
    The loop calls recv(512) repeatedly rather than assuming that the response arrives in one call.
  2. 2
    The condition if not chunk detects the empty return that signals the end of the response.
  3. 3
    Each non-empty chunk is decoded before printing because the socket provides raw bytes.
  4. 4
    The end='' argument prevents print() from adding an extra newline after every chunk, so the chunks appear continuously.
  • Start a loop because the complete response may require multiple receives.
  • Request up to 512 bytes from the socket.
  • Stop when the socket returns no more data.
  • Decode the byte chunk into a string and print it without adding another newline.

From Bytes to Text

The data returned from the socket is binary data represented as bytes. Those bytes are not yet the readable text that should be printed. Calling decode() converts each received chunk into a string. The program therefore has two distinct representations: bytes while data is moving through the socket, and a string after decoding for display.

Received bytesraw binary datadecode()conversion stepReadable stringtext for printingconvertproduce
How does raw byte data received from the socket become text for the program to print?
RepresentationWhere it appearsWhat the program does
BytesData returned by recv()Decode before printing
StringDecoded response chunkPrint as readable text

The socket stage and display stage use different representations of the response.

Edge case

Decoding is a separate step from receiving. If the program tries to print the received byte chunk as though it were already text, it has skipped the conversion required by this process. The decoding step must occur on each chunk before it is printed.

Completing the Socket Lifecycle

Worked example

A complete request and response cycle

Trace the order of operations for a client that retrieves the root resource from a server on port 80.

  1. 1
    Create

    Create a socket object using Python's socket module.

  2. 2
    Connect

    Connect the socket to the remote server on port 80. The TCP three-way handshake establishes the reliable channel.

  3. 3
    Format

    Build a GET request containing the resource path, the HTTP version, and the terminating sequence \r\n\r\n.

  4. 4
    Encode

    Convert the request string to bytes with encode() because sockets send bytes.

  5. 5
    Send

    Send the encoded request through the connected socket.

  6. 6
    Receive

    Call recv(512) repeatedly. Decode and print every non-empty chunk.

  7. 7
    Finish

    When recv() returns an empty result, stop the loop and close the socket.

Result

The client has followed the complete socket-based HTTP exchange: connection, formatted request, byte transmission, chunked reception, decoding, output, and closure.

python

The try and finally structure gives the socket a clear lifecycle. It is created first, connected before use, used for sending and receiving, and closed after the communication work is finished. The receive loop ends because the server has no more data to provide, not because the client assumes that one recv() call contains the whole response.

Mistakes to Avoid

  • Common mistake

    Sending the request as a string instead of bytes.

    What it looks like
    sock.send(request)
    Why it fails

    Python sockets work with bytes, while the request was constructed as text.

    Do this instead

    Encode the request before sending it: sock.send(request.encode()).

  • Common mistake

    Leaving out the final blank line.

    What it looks like
    GET / HTTP/1.0\r\n
    Why it fails

    The HTTP request needs \r\n\r\n to signal the end of its headers.

    Do this instead

    End the request with two consecutive line endings.

  • Common mistake

    Calling recv() only once.

    What it looks like
    chunk = sock.recv(512)
    Why it fails

    The response arrives as a stream and may require multiple chunks.

    Do this instead

    Place recv(512) inside a loop and continue until the return is empty.

  • Common mistake

    Printing received bytes without decoding them.

    What it looks like
    print(chunk)
    Why it fails

    The received chunk is raw binary data rather than the readable string expected for display.

    Do this instead

    Decode each non-empty chunk before printing it.

  • Common mistake

    Adding a newline after every received chunk.

    What it looks like
    print(chunk.decode())
    Why it fails

    print() adds its own newline, which can insert extra line breaks between chunks.

    Do this instead

    Use end='' when printing chunks continuously.

Practice the Trace

MEDIUM

Explain the journey of data in this sequence: a request string is created, encode() is called, the request is sent, recv(512) returns twice with data, and then recv(512) returns an empty result. Identify what representation is used at each stage and state exactly when the loop stops.

Hints
  • The socket sends bytes rather than the original request string.
  • Each non-empty received chunk must be decoded before printing.
  • The empty return from recv() is the stopping signal.

What do you think happens?

A response is larger than 512 bytes. Will the first recv(512) call necessarily contain the complete response?

  • Yes, recv(512) always waits for the complete response

  • No, the response may arrive in multiple chunks

  • Yes, because HTTP responses have a fixed size

Reveal answer
Answer:

No, the response may arrive in multiple chunks.

The response arrives as a stream, and recv(512) retrieves up to 512 bytes at a time. The program must continue receiving until an empty result signals that the response is finished.

Key Takeaways

  1. 1
    A connection to port 80 creates a reliable channel through a TCP three-way handshake.
  2. 2
    An HTTP GET request must be encoded into bytes and terminated with \r\n\r\n.
  3. 3
    recv(512) retrieves up to 512 bytes at a time, so responses must be handled in a loop.
  4. 4
    Each non-empty byte chunk is decoded into a string before it is printed.
  5. 5
    An empty result from recv() signals that the server has finished sending, after which the socket should be closed.

At a glance

  • Sockets provide the communication channel, while HTTP defines the request format sent through that channel.
  • The request changes from a Python string to bytes through encode() before transmission.
  • The response must be received incrementally because recv(512) returns only up to 512 bytes per call.
  • Each received chunk changes from bytes to text through decode() before printing.
  • The empty return from recv() ends the receive loop, and the socket is then closed.

Related Topics

Related concepts

More concepts starting with H