[MUD-Dev] MMORPG/MMOG Server design

[J C Lawrence]

On Mon, 17 Feb 2003 11:55:22 -0600
Weston Fryatt <wfryatt at muuf.com> wrote:

> What is the best design for an MMORPG Server?

There isn't one best.

There are many, each for a slightly different set of requirements or a
slightly different set of value trade offs.

You get to pick.

> Single Massive server? The whole game run on one multi-processor
> machine with gigs of ram.

You're starting from the wrong base.  First start with your
requirements.  There are a few requirements and a bunch of metrics which
you'll have to decide before that:

  0) Uptime management: redundancy, failover, etc.

  1) What must the server DO?  (Define that as tightly as you can in
  every manner you can think of and then some)

  2) What's your working set size?

  3) What is the rate of page migration thru your working set?

  4) What are your synchronisation domains within your world design and
  working set?  (Actually a useless question except that it makes you
  count the real working set size and not just a way-too-small guess)

  5) How large is your connection pool (# of sockets)?

  6) What is your transaction rate per socket
  (open(2)/read(2)/write(2)/close(2))?

  7) Method of managing/approaching the C10K problem?

  8) What is your process/threading model?  (And thus required context
  overhead, context shift SMP CPU cache invalidation due to shared
  memory, etc)

  9) What is your data rate into and out of the server?

  10) What are the processing requirements of your various support
  systems (eg billing, authentication, tracking, etc)?

The list goes on quite a bit more, but I'll leave that to you.

Don't forget to calculate basic required BUS speeds and bounding boxes
on MIPS per game-operation that will be required by your system -- that
will give you the clear sanity check as to what is possible, or not, and
what your hardware must be capable of for a given approach.

> Server Groups: A group of servers runs a sub-set of the game world
> with a max user count of 2000. Each server group would be a unique
> part of the game world.

> Redundant Servers: Multiple Servers doing the same job all persisting
> the game world/game state into a common database cluster server. Each
> server has only a specific task. (ie: map server, login server, game
> logic server, npc server, etc) You may have 10 map server running, 35
> game logic server, etc running.

> I'm leaning towards the Redundant Server idea, But is this the best
> idea?

There are others.

In the typical clustered server case the cluster is not homogenous, but
rather several are specialised.  Typically there's a connection proxy
which handles all user connections and represents them internally to the
currently appropriate internal box.  Then there's authentication,
billing services, CMS and all the rest of that crap which usually take a
box or three.  Behind all that you have something that is "running the
game".  Within that space there are multiple approaches, all with large
tradeoffs.

A common approach is to segment the space.  Breakdowns I've seen:

  -- presentation, logic, backing store

  -- geographical segmentation of the game world into "zones"

  -- player processing, automation processing, backing store

Segmentation of the backing store away from processing leads to high
transactional rates, and coherency issues should multiple nodes attempt
to process the same data.  Resolving those problem can rapidly lead to
code and design complexity.

Segmentation into "zones" makes for abrupt and very opaque zone
transitions.  This can be jarring to the player and requires careful
design and implementation to not expose exploit paths.

On the homogenous back end cluster side there are several different
approaches, each with tradeoffs in hardware expense, code complexity,
etc.  You can go for SGI-style CC:NUMA clusters with fully virtually VM
and CPU resources such that processes and VM blocks freely migrate about
the cluster according to your load distribution algorithms.  Without
significant support from your system vendor and an experienced
development team this can be expensive and difficult to get right, but
works beautifully when it does.

You could go a bit lighter and leave the data stationary and migrate the
process pools (cf Mosix) about and live with the MPI overhead for the
cases where data is remote or processes need to coordinate.  This could
be attractive depending on your working set size and the semantics of
your shared/common working set size and transaction rate (ie remote
invalidations).  This could be wonderful or a huge pain as well,
depending on how you handle flash crowds.

There are quite a few other cluster models, but that's enough to get
thinking with.

But the key point is that these are not the decisions to be made now.
What's the best design for a car?  One that does what you need.
Ferraris are great at going fast and driving your insurance bills thru
the roof.  If that's what you need, get one.  They don't do so well at
carrying the weekly shopping back from the grocery store in rush hour
traffic, or a load of lumber and tools for building a deck behind your
house while somehow also managing to pay for the kid's school bills.  If
that's what you need then something not-Ferarri is advised.  A Ferrari
is probably also not advised if you live in a high crime area, or have
almost no paved roads...

Figure out your requirements.

Figure out what you're doing first, then design to satisfy those
requirements.  Architecture is almost entirely a question of context.  A
decent way to approach that is to start out by list system behaviours.

  The system must....<something>

Segment that list into three sets:

  1) The project is dead if it doesn't do...

  2) The project is devalued if it doesn't do...(and define the impact
  in detail)

  3) The project is improved if it does... (and define the scope and
  bounds of the improvement carefully).

Fight hard to have an empty list for #1.  It never will be empty (or you
don't have a job), but try and make it empty.

Now try and make #2 even emptier by pushing things into #3.

Repeat aggressively.

Figure out the absolute minimum architecture that can handle #1 (ie the
one that requires the fewest decisions, the fewest definitions, the
fewest control points, etc).

Work over it a few more times making it simpler.

Now go back and see what you can migrate from list #1 to list #2, and
then work back over your architecture again.

Repeat until it hurts and all your sentences defining the architecture
are very simple and contain as few adverbs and adjectives as possible.
Keep going until your paragraphs are as short and obvious as your
sentences.

Review that against your list in #2 and see if you can't make your
approach even cleaner by accomodating items from that list.  It sounds
counter-intuitive, but it usually works that way.  A frequent sign for
me for when I hit the sweet spot with a design is when my design
definition suddenly collapses both in volume and complexity while the
feature set rises dramatically.

Try and shove as much as you can from the #2 list into #3.

Repeat etc ad nauseum.

--
J C Lawrence
---------(*)                Satan, oscillate my metallic sonatas.
claw at kanga.nu               He lived as a devil, eh?
http://www.kanga.nu/~claw/  Evil is a name of a foeman, as I live.

_______________________________________________
MUD-Dev mailing list
MUD-Dev at kanga.nu
https://www.kanga.nu/lists/listinfo/mud-dev