[MUD-Dev] Modular Design Issues RFC

[Russ Lewis]

Ryan Rhodes wrote:

> (snip) In short, I'd really appreciate any expert java opinions.

I can't say I'm a Java expert, but I think I'm working on something
(in C++, unfortunately) that would apply to many of the issues you're
discussing here.  Basically, it's a program library (that has
aspirations to become a full OS someday) that deals well with many of
the issues of modularity and parallelism you discuss here.  Or, more
accurately, the design looks like it could...I'm in the process of
developing it as my B.S.  senior project.

I hope this doesn't come across as an ad...I simply have been working
on a solution for what I've wanted for a long time, and now that I'm
developing it, I'm excited to get the word out. :) Please excuse the
long post and evangelical fervor....

I call this library/OS demingOS.  An anonymous CVS tree is available
for those who are interested
(:pserver:anonymous at cvs.deming-os.org:/cvs, project module name is
deming), as well as a 6-month out of date website at
www.deming-os.org.  (Sorry, the website is SOOO out of date....if
people tell me they're interested and actually looking, I'll make it a
priority to bring it up to speed.)

> What is modularity?
>
> I've read a lot of the posts on modular design and the issue seems to
> get fuzzier the farther you get down the thread.  I've seen everything
> from separate threads for areas to separate processes for multi-world
> systems.  As best I can divide the issues they seem to center around
> three ideas of modularity.
>
>   1. Logical
>   2. Distributed
>   3. Threaded
>
> We modularize in the logical sense to divide the problem into more
> understandable parts, to encapsulate data, for code reuse, for
> composition.  Our system has a complete object model encapsulating
> every aspect of them game into objects.  The system is divided along
> clear interfaces.
>
>   Question: Is it not true, that while we may divide our system into
>   threads and processes to deal with other issues, we do not divide
>   our system into threads and processes to increase the logical
>   modularity?

I'll step in here with my design, if you'll excuse me.  In demingOS,
you, the programmer, break your program down into objects that I call
"Components."  Each component is basically an object that encapsulates
a certain portion of your product.  You might have one component for
your network code, one for your game engine, one for you AI, etc.
There could be, conceivably, thousands of components running at once.

However, they do not each have thread(s) assigned.  Instead, each is
driven by an event-based messaging model where shared threads dispatch
the messages.  These messages are transmitted through "interfaces,"
which are defined links between components.  Each interface models a
different communication type, or a different instance of
communication, and they are modular.  Thus, you could define a "player
interface" and have both human players over the network and NPCs on
the local machine connect through the "player interface."  The
difference is that the interfaces for human players come from your
network component (whose job it is to translate network traffic into
this interface and vice-versa), and the interfaces for the NPCs come
from a component that processes AI decision making.

I will explain more as you continue...

> Modularity, in terms of distributing our system among multiple
> processes seems to have been done for two primary reasons in the
> posts.
>
>   1. To allow components to be processed on multiple machines

>From the very start, demingOS was designed to work in a distributed
environment.  While my current senior project only supports local
processes, I have already designed (in my head) a "router" design that
will be able to route any demingOS interface through any type of
network (even email!) to another demingOS kernel.  You simply need to
implement a router component for the proper network type; surely
TCP/IP would be my first router.

This means you can do any type of distributed processing.  You could
put your network interface running on one machine, which would also be
configured with a firewall.  The whole rest of your network would be
running on a LAN behind the firewall.  If the component that runs the
game engine was made modular, then you could have multiple engine
servers, each running one area of your game world.  The player
interfaces would just be routed interfaces from the firewall/network
server to the appropriate engine components.  The NPCs could be
running on one or more other, dedicated servers, also connecting to
the engine.  And so on...

The beauty of this system is that while deming knows that particular
interfaces are routed to remote components, it can treat them just
like they are local.  Thus, while you are developing your game, you
will probably only have one server that performs all of the tasks.  As
your player load increases, you can gradually move certain components
to new servers with absolutely zero changes in your code.  Just change
the configuration to route the interface over a network, instead of
doing it locally.

The only restriction is that one instance of a component type cannot be
spread across multiple servers; however, you can create as many instances
of any component as you like, either on a single server, or spread across
multiple servers.

>   2. To allow components to be hot bootable

Since all programs are "components," demingOS naturally has the
ability to dynamically load, configure, and connect them.  It's up to
you to make sure that the game world is distributable, and that you
have procedures for moving data and players from one server to another
when one starts up or shuts down.

> When I say hot bootable I mean to allow a reboot of the Data Backend
> without a reboot of the game, or the game without the server and
> vice versa.

Better yet, once you implemented code to dynamically shift load
amongst servers, you can even do code upgrades online.  Simply install
the new component on a server (deming automatically tracks multiple
installed versions of a component-no need to overwrite the old
version).  Then you take down that particular server; your code
automatically shifts the load to other ones.  Then you boot up the new
version of the server.  Since you are still using the same interfaces
as before (even though your component is a new version), your new code
plugs into the game world, and the load shifts back onto it.  Repeat
as necessary until you have upgraded all of your servers.  The same is
true, of course, of all components, including the NPC/AI and network
components.

>   Question: Are these the only two reasons to divide a system into
>   separate processes?

I would also argue that splitting code up enforces good coding
techniques because you can implement some fairly strict memory
protection.  In deming, I go so far as to say that it's impossible to
share memory in any form; otherwise my scheme of being distributable
over a network wouldn't work.

> I've seen three primary candidates in the posts for a separate process.
>
>   1. Game Servers
>   2. Connection Servers
>   3. Data Servers
>
> The game would be distributed only if you wanted to allow for worlds,
> or a world, distributed over multiple systems.  Distributed servers to
> allow for multiple login sights and connection handling over multiple
> systems.  It would seem as if the database should, at least logically,
> appear as if it's on a single machine but could easily be distributed.
> All three components might even run on a single system and receive the
> hot booting benefits of a distributed architecture.
>
>   Question: Are these the only three logical subsystems that benefit
>   from distribution?

In demingOS I would break it up even further...though it might not
make sense in a more traditional OS.  Perhaps your file I/O would be a
dedicated component...or maybe you have components that automatically
connect to the game engine, get data summaries, and convert them to
HTML for "near realtime" game statistics.  Even more likely, the
engine itself is probably many different components: weather, mapping,
combat, economics, etc...

> How many Threads???

My favorite thing of deming is that this is absolutely a non-issue.  I
mentioned above that deming does not assign threads to specific
components.  Instead, all threads are shared amongst the components.
When a message arrives (through an interface) on a component, then a
kernel thread calls a callback function (called a "handler") for that
message type, temporarily giving the thread to that component.  When
the handler completes and returns, the thread returns to the global
pool.

This system is extremely flexible.  In a single processor arrangement,
one thread will do the vast majority of the work.  The only time that
a second thread will be used (in the entire OS!!!) is when the first
one blocks and requires parallel processing of some type, or when a
handler in the first thread takes too long in returning.  In either
case, a second thread is spawned and the kernel runs multithreaded
until the condition clears.  demingOS will spawn as many threads as it
needs automatically, and automatically shut down threads if they have
been unused and unneeded for a certain amount of time.  In a
multiprocessor system, deming spawns one thread for each processor,
and again they handle the vast majority of the workload.  deming just
spawns new threads as necessary to keep the processor active and to
ensure that a single long handler (or one caught in a spin loop)
doesn't monopolize the system.

Moreover, there is no natural limitation on how many threads might be
"dispatched" into one component at a time.  If you are running the
game engine on a 4-way SMP machine, you should expect that, at most
times, you will be handling 4 different messages (or more) at once FOR
THE SAME COMPONENT.  Thus, each component is naturally multithreaded,
and naturally expands its multithreadedness to fill the available
processors and processor time.

Of course, deming's design includes the ability to control
multiprocessing; you can set "dispatch limits" that control how many
messages can get dispatched to you at once.  If you want some type of
message to be serialized (handled in a linear fashion), set its
dispatch limit to 1, and deming will not dispatch a new message of
that type to your interface until you have finished processing the
last one.


> Having the logical modularity and distribution issues already covered,
> there seems to be left only 4 reasons for threads in a system.
>
>   1. To block for Data Base I/O

File I/O is handled by a component in deming.  To write, you send it a
message containing the data.  There's no need to block, waiting for it
to actually accomplish it.  However, if it is critical (a game state
backup, for instance), you can block until you get a reply on the
message, but this block only blocks the handler you are in.  Other
handlers on the same component, even handling an identical message
type, do not block.

To read, you send the file I/O component a message that you want to
read, and it sends you a series of messages, each of which contains a
block of data from the file (or, if the buffer is large, perhaps the
entire file in one message).  If you need to have the whole file at
once, then write a class to buffer the file as the file I/O component
sends you parts of it, then send yourself a "file ready" message when
you have the whole thing.

>   2. To block for Socket Input

This can work just like File I/O.

>   3. To block for incoming connections

Again, this is just like File I/O, except that (after setup) it's
pretty much a one-way connection.  When a new connection comes in, you
get a message about it, and you can then set up an interface to the
network component to handle that connection.

>   4. To simulate time and action in the world

I'm not sure what you're speaking about here.  Is this just timer
messages?  You could easily build the kernel to send a component
"tick" messages at various times....

> (snip questions about MUD programming.  While I'd like to write a MUD
> someday, I haven't yet).

> Assuming that our architecture modularizes the system logically.
> We've used distribution for the reasons stated above and threads to
> eliminate I/O blocking from halting the CPU.  Beyond that, using
> additional threads to simulate time and process game events just slows
> down the system.
>
>   Question: Is it not true then that to divide any non-blocking
>   sequential piece of code into multiple threads only degrades
>   performance.  Threading is used purely to prevent the CPU from
>   waiting on I/O?

Generally, this is my understanding (a key motivation for the shared
thread design in deming).  However, to be fair, in an SMP system,
multithreading can occasionally help things out, as it allows more
things to be handled simultaneously (i.e. if you have data from two
different network connections, two processors can process it in
parallel if you have multiple threads).  A lot of this performance
boost can be lost, however, if you don't have good thread
coherency...that is, data that is in the cache on processor A won't be
in the cache on processor B, so a thread using data that A has will
run more slowly on B until the cache is updated.

>   Question: Am I correct in assuming that most game systems out there
>   use a single threads "Game Loop" to simulate time and, indirectly,
>   trigger the parsing and processing of all game events?

Can't speak to that, other than I know that the Diku core (the only source
I've seen) not only uses a single game thread, it handles *everything* in
that one thread.

(snip the rest, that I can't really speak to...)

P.S. The idea of routing interfaces over a network is flexible...one
of my long-term goals is to have developers use demingOS as the base
for client code.  Thus, instead of having a firewall/network
translator component, you would have the client software make a
demingOS router connection over the network either to a load-balancing
firewall, or directly to your game engines.  Then we stop seeing this
as a "network game" and start seeing it as a "deming game" running
over a network.

P.P.S. I'm building deming as a virtual OS on top of other
OS'es...this means that it is extrememly portable; I don't have to
write compilers, drivers, or any other low-level stuff for a new
platform.  Instead, I write an "atom" (kind of like a kernel for the
kernel) that translates a small set of standard deming calls (like
GetLock(...)) into operating-system specific calls.  Similarly, only
standard deming types are used; all atoms must include a typedef for
int8, uint32, etc.  To port your code to a new architecture, just take
your code, which conforms to the deming standard, to a new platform
and recompile it using your favorite compiler, and linking to the
deming atom library for that platform.

Hope this all was interesting to someone and annoying to no one.

Russ Lewis


_______________________________________________
MUD-Dev mailing list
MUD-Dev at kanga.nu
https://www.kanga.nu/lists/listinfo/mud-dev