[MUD-Dev] [Tech] socket slowdown solved

[Chris Gray]

Back in August (I was checking the archives), under the thread
"Ethernet NICS, maximum connections..mud testing", I had indicated a
performance problem I was having with communication between my server
and client, when running on the same Linux machine. Very little of
the CPU time was used, but there was no explanation for the slowdown.

Well, yesterday was the day I finally got a small test program to
display the same behaviour (it relates to mixing requests with replies
with requests without replies). I emailed it to myself at work to try
out on some other UNIX variants. Same result on all of them! It turns
out that today I was also chasing UDP performance problems for work
things, and in my investigation, I came across the TCP_NODELAY option
to setsockopt for the TCP protocol layer. Since the work stuff is UDP,
not TCP, that was no help, but I decided to try it on my test client
and server. Voila! Drastic improvement in performance. I've just put it
into the real server and client here at home, and the speedup is around
a factor of 10! At last!

So, I'm posting this to the list in case anyone else runs into this
same problem, and doesn't find the answer in a handy book. Note that
the Linux setsockopt man page does mention this, and my Linux at least
doesn't have a 'udp' page in section 4p (no 4p at all!). SunOS4 had
one dated 1987 that explains this, so its not a new thing.

So what is going on? There is a "Nagle algorithm" in the TCP code on
(most? all?) UNIX boxes. It is intended to cut down on the sending of
small packets on the network, thus reducing network load and system
load. It does this by waiting a small amount of time (I'm betting its
20ms on this box) to see if there will be another packet following soon.
If there is, it combines them. There appears to be some heuristics in
there, since testing with a program that *just* sends small packets
sees no slowdown (or perhaps there is one, but because the server and
client are not synchronizing, its effect is masked). A program that
does only send/reply pairs also is not slowed down, perhaps because
the TCP code recognizes the situation and sets some flags or something.
However, if your traffic consists of alternating requests with replies
and requests without replies, the slowdown shows up.

Doing

    on = 1;
    setsockopt(socket, IPPROTO_TCP, TCP_NODELAY, &on, sizeof(int));

tells the system to disable that heuristic for that socket. Note that
this is not related to things like FIONBIO which make a socket a
non-blocking socket. In my simple test program, I only needed to do
this in my client, but in the real thing, I needed it in both the
client and the server.

Since this does put an additional burden on the machine and network,
it may be advisable to only do it for local connections (where the
client and server are on the same machine), or where the connection
is *very* fast. Since I see a delay of 20ms here, and my fastest 'ping'
time to another machine is 30ms, I probably should do it only for local
connections. Also, in my case it only matters at all when the client
is being used to source 30,000 lines of MUD-code to built the world.

Now I can get on with real work on my MUD!

--
Chris Gray     cg at ami-cg.GraySage.Edmonton.AB.CA