[DGD] DGD/MP & OpenMosix revisited?

[Greg Lewis]

On Mon, Feb 09, 2004 at 02:28:03PM +0100, Felix A. Croes wrote:
> Ragnar Lonn <prl at gatorhole.se> wrote:
> > DGD/MP sounds good but we also need some kind of clustering in order to
> > create really large game environments. For operational and cost reasons
> > you want to be able to host a service on many cheap servers, rather than
> > a single gigantic one. RAID-based storage is a good analogy.
> 
> Here is where I start to disagree.
> 
> Every MMORPG since Ultima Online has used clusters of cheap servers.
> I never learned the exact details, but I believe that UO used (and
> perhaps still uses) more than a dozen machines for each shard.
> 
> Multiply that with the number of shards and you'd very, very much prefer
> a single big machine rather than a cluster of many cheap servers, or
> as it turned out in the case of UO and most MMPORGs that came after,
> a cluster of clusters of many cheap servers.  First, there is the
> maintenance cost of all these servers.  Compared to the typical
> mainframe, PC hardware is rather unreliable, and with a sufficiently
> large pool of active machines, you are pretty much going to have
> hardware failures every week.  Replacing PC parts is cheap, but paying
> someone to take care of it is not.  Replacing a failed disk is easy,
> but what do you replace if your machine has intermittent failures?  In
> my experience those repairs can drag out over several days, or even
> weeks.

Here is where I start to disagree :).  The trend for large
computationally intensive tasks is not a single large machine, for
several reasons.  Glance at the top 500 super computer list and you'll
find it dominated by clusters, especially at the top end.

Your argument regarding maintenance assumes low quality generic
brand parts.  You'll find that most decent cluster manufacturers will
use good quality components.  MTBF is important not only to the
customer buying the cluster but also to the company selling it (which
will have to pay for shipping and replacement parts when the customer
RMAs a faulty node).  I'd argue that a good quality cluster from a
reputable company is now seeing similar MTBF numbers to similar
size large machines, at a fraction of the cost.

> Then there is the software maintenance cost.  All of these machines
> have an operating system which must be installed and kept up to date.
> Any OS problem is potentially multiplied by the number of machines.
> The hardware could perhaps be managed by a single person, the software
> can not.

Sorry, but I have to disagree with this entirely.  In a cluster it is
customary to manage the operating system for the nodes as images.  In
MUD cluster I'd expect there to be two production images, one for the
standard compute nodes which are running the game and one for some
I/O nodes which are managing state dumps/database transactions.  You
do _not_ manage the OS for each individual node unless you're a masochist.

State of the art is to clone these images onto the relevant nodes over
a management network (avoiding congestion on the main interconnects)
using multicast.  The nodes themselves have network bootable NICs using
PXE or have something like etherboot flashed onto the NIC ROMs.

It is simple for one person to manage the OS images in this situation
and you can literally have a 1,000 node cluster cloned and running in
about 5 minutes.

Not only that, but any cluster of significant size is running some sort
of monitoring software which will allow this same person to monitor the
nodes (loads, temperatures, whatever).  Quite possibly with some automated
responses when a problem (e.g. overheating) is detected (e.g. migrate
the tasks on that node to another node and shutdown).

> Add to this the cost of designing your MUD to be distributed (which is
> not trivial), the cost to the MUD itself of being distributed (some
> features that you'd really like to have are just not going to be
> possible), the cost of making sure that the MUD software keeps working
> as intended in this distributed environment, the cost of synchronizing
> MUD software updates across all machines, and probably quite a few more
> things that I didn't think of -- at this point, you are paying a lot of
> people to manage your overhead of cheap hardware on a fast network.

Designing a distributed MUD is certainly a challenge, at least we agree
on this one :).  However, this is where a transparent process migration
technology like OpenMosix of bproc can help simplify things.

> MUDs are like databases.  The ultra-fast, ultra-large, ultra-reliable
> databases don't run on a network of cheap hardware.

Not true anymore.  Oracle has certified a number of Linux cluster based
solutions in the last year or two and feature them at their solution
centre to show prospective clients.

-- 
Greg Lewis                          Email   : glewis at eyesbeyond.com
Eyes Beyond                         Web     : http://www.eyesbeyond.com
Information Technology              FreeBSD : glewis at FreeBSD.org

_________________________________________________________________
List config page:  http://list.imaginary.com/mailman/listinfo/dgd