workload simulation (was: Re: [MUD-Dev] MMORPG/MMOG Server design)

[Mike Shaver]

On Feb 23, Damion Schubert wrote:

> This is fine, provided that movement is your bottleneck.  However,
> over the years, I've seen different parts of the game be the slow
> one.  Monster AI and pathfinding, as well as spawning, can have
> huge impacts on CPU time and bandwidth, but if your load testing
> is just putting faux characters on an empty world, that won't be
> factored into your results.
 
> Similarly, some systems (such as guilds) work fine with no
> apparent server slowdown when there are only 2-3 guilds with 2-10
> people in them (as there usually is when the coder is testing his
> own work), but fail cripplingly when put in a (hopefully) beta or
> (less optimally) live environment, and two dozen 200 person guilds
> spring up.
 
> Thirdly, CPU and bandwidth varies largely based upon whether it's
> day one or day one hundred.  On day one, you have more people in
> an area touching fewer spawners - this is a quite different
> pattern than the rest of the life of the game (aside from wipes).

My first reaction to all of this is "log and replay": as soon as you
have, say, 1500 people logged into your server once, you can use
data captured from that session to test the performance of later
updates.  There are lots of systems that work this way, from
log-driven web-server benchmarking to TCP reconstruction for IDS
tools to system-call replay work like Michael Elizabeth Chastain's.
It seems that one already has a great set of data from testers in a
lot of cases, and that -- barring protocol or system changes that
outstrip your ability or desire to migrate the data set -- you can
use it to validate future builds.

  (Does the Shadowbane recorder do enough of this that you could
  turn it on its head and spit data at the server, instead of
  replaying the server's responses?  The Age of Empires
  synchronized-RNG/game-replay stuff also sounds like it might be
  interesting prior art for someone heading down this path.)

If nothing else, you could use it to run the current build under a
profiler or debugger, to examine trouble spots at your leisure.  I
imagine that such servers can process a fair amount of data, but
disk is pretty cheap, and it's _very_ cheap when compared to
programmer time.

As far as "balance testing" goes, mining that data set so that you
can do before/after comparisons of "player X uses power Y on player
Z" in terms of damage/hit rate/XP gain/etc. to sanity-check design
changes also seems like fertile ground.  It's not a silver bullet,
by any means, and I'm sure we can all construct cases that are
really hard to test this may, but having a tool that automates even
30% of the process lets you spend that much more human-time on the
parts of balance that are more resistent to mechanical analysis.

Of course, this whole thing is a lot easier if your game's
characteristics are data-driven, rather than open-coded into the
combat parts of your server, but who does that sort of thing in the
year 2003?  =)

Mike

_______________________________________________
MUD-Dev mailing list
MUD-Dev at kanga.nu
https://www.kanga.nu/lists/listinfo/mud-dev