How to Use TCP Effectively

by Warren Young

Newcomers to network programming almost always run into problems early on where it looks like the network or the TCP/IP stack is munging your data. This usually comes as quite a shock, because the newcomer is usually told just before this that TCP is a reliable data transport protocol. In fact, TCP and Winsock are quite reliable if you use them properly. This tutorial will discuss the most common problems people come across when learning to use TCP.

Problem 1: Packets Are Illusions

This problem comes up in various guises:

• "My client program sent 100 bytes, but the server program only got 50."
• "My client program sent several small packets, but the server program received one large packet."
• "How can I find out how many bytes are waiting on a given socket, so I can set up a receive buffer for the size of the packet?"

I think that understanding this issue is one of TCP/IP's rites of passage.

The core concept that you must grasp is that TCP is a stream protocol. This means that if you send 100 bytes, the receiving end could receive all 100 bytes at once, or 100 separate single bytes, or four 25-byte chunks. Or, the receiver could receive that 100 byte block plus some data from the previous send and some from the succeeding send.

So, you ask, how can you make a program receive whole packets only? The easiest method, in my experience, is to prefix each packet with a length value. For example, you could prefix every packet with a 2-byte unsigned integer that tells how long the packet is. (See problem 2 for advice on properly sending integers across a network.) Length prefixes are most effective when the data in each protocol packet has no particular structure, such as raw binary data. Here is some C++-like pseudocode showing a common algorithm for dealing with length prefixes:

    int new_bytes_read = 0;

    // Read in the prefix int packet_size = 0, prefix_bytes_read = 0;
    const int prefix_size = 2; do {
        char temp_buffer[prefix_size]; new_bytes_read = recv(sd,
        temp_buffer,
                prefix_size - prefix_bytes_read, 0);
        if (new_bytes_read >= 1) {
            for (int i = 0; i < new_bytes_read; ++i) {
                packet_size <<= 8; packet_size |= temp_buffer[i];
            } prefix_bytes_read += new_bytes_read;
        } else {
            // Handle the error...
        }
    } while (prefix_bytes_read < prefix_size);

    // Allocate the buffer
    char* packet_buffer = new char[packet_size];

    // Read in the packet
    int packet_bytes_read = 0;
    do {
        new_bytes_read = recv(sd, packet_buffer + packet_bytes_read, 
                packet_size - packet_bytes_read, 0);
        if (new_bytes_read >= 1) {
            packet_bytes_read += new_bytes_read;
        }
        else {
            // Handle the error...
        }       
    } while (packet_bytes_read < packet_size);

Notice how we loop on recv() for both the length prefix as well as for the remainder of the packet.

Another method for setting up packets on top of a stream protocol is called "delimiting". Each packet you send in such a scheme is followed by a unique delimiter. The trick is to think of a good delimiter; it must be a character or string of characters that will never occur inside a packet. Some good examples of delimited protocols are NNTP, POP3, SMTP and HTTP, all of which use a carriage-return/line-feed ("CRLF") pair as their delimiter. Delimiting generally only works well with text-based protocols, because by design they limit themselves to a subset of all the legal characters; that leaves plenty of possible delimiters to choose from. Note that several of the above-mentioned protocols also have aspects of length-prefixing: HTTP, for example, sends a "Content-length:" header in its replies.

Of these two methods, I prefer length-prefixing, because delimiting requires your program to blindly read until it finds the end of the packet, whereas length prefixing lets the program start dealing with the packet just as soon as the length prefix comes in. On the other hand, delimiting schemes lend themselves to flexibility, if you design the protocol like a computer language; this implies that your protocols parsers will be complex.

There are a couple of other concerns for properly handling packets atop TCP. First, always check the return value of recv(), which indicates how many bytes it placed in your buffer it may well return fewer bytes than you expect. Second, don't try to peek into the Winsock stack's buffers to see if a complete packet has arrived. For various reasons, peeking causes problems. Instead, read all the data directly into your application's buffers and process it there.

Problem 2: Byte Ordering

You have undoubtedly noticed all the ntohs() and htonl() calls required in Winsock programming, but you might not know why they are required. The reason is that there are two major ways of storing integers on a computer: big-endian and little-endian. Big-endian numbers are stored with the most significant byte in the lowest memory location ("big-end first"), whereas little-endian systems reverse this. (There are even bizarre "middle-endian" systems!) Obviously two computers must agree on a common number format if they are to communicate, so the TCP/IP specification defines a "network byte order" that the headers (and thus Winsock) all use.

The end result is, if you are sending bare integers as part of your network protocol, and the receiving end is on a platform that uses a different integer representation, it will perceive the data as garbled. To fix this, follow the lead of the TCP protocol and use network byte order, always.

The same principles apply to other platform-specific data formats, such as floating-point values. Winsock does not define functions to create platform-neutral representations of data other than integers, but there is a protocol called the External Data Representation (XDR) which does handle this. XDR formalizes a platform-independent way for two computers to send each other various types of data. XDR is simple enough that you can probably implement it yourself; alternately, you might take a look at the Libraries page to find libraries that implement the XDR protocol.

For what it's worth, network byte order is big-endian, though you should never take advantage of this fact. Some programmers working on big-endian machines ignore byte ordering issues, but this is bad style, if for no other reason than because it creates bad habits that can bite you later. Other interesting trivia: the most common little-endian machines are the Intel x86 and the Digital Alpha. Most everything else, including the Motorola 680x0, the Sun SPARC and the MIPS Rx000, are big-endian. Oddly enough, there are a few "bi-endian" devices that can operate in either mode, like the PowerPC and the HP PA-RISC 8000. Most PowerPCs always run in big-endian mode, however, and I suspect that the same is true of the PA-RISC.

Problem 3: Structure Padding

To illustrate the structure padding problem, consider this C declaration:

    struct foo {
        char a;
        int b;
        char c;
    } foo_instance;

Assuming 32-bit ints, you might guess that the structure occupies 6 bytes. The problem is, many compilers "pad" structures so that every data member is aligned on a 4-byte boundary. Compilers do this because modern CPUs can fetch data from properly-aligned memory locations quicker than from nonaligned memory. With 4-byte padding on the above structure, it would actually take up 12 bytes. This issue rears its head when you try to send a structure over Winsock whole, like this:

    send(sd, (char*)&foo_instance, sizeof(foo), 0);

Unless the receiving program was compiled on the same machine architecture with the same compiler and the same compiler options, you have no guarantee that the other machine will receive the data correctly.

The solution is to always send structures "packed" by sending the data members one at a time. Or, you can force your compiler to pack the structures for you. Visual C++ can do this with the /Zp command line option or the #pragma pack directive, and Borland C++ can do this with the -a command line option. Keep the byte ordering problem in mind, however: if you send a packed structure in place, be sure to reorder its bytes properly before you send it.

Conclusion

The moral of the story is, trust Winsock to send your data correctly, but don't trust that it works the way you think that it ought to!

Copyright © 1998, 1999 by Warren Young. All rights reserved.