[MUD-Dev] [TECH] Peer-to-peer downloads for MMO games

[Olof Ekstrom]

Abstract: 

  A suggestion for use of peer-to-peer technology to limit network
  demands on servers in massively multiplayer scenarios with large
  download volumes.

Keywords: 

  Client side caching, peer-to-peer, patching, network bandwidth
  Content: Suggested service outline, request for input

I'm sure readers of this list are more knowledgeable than myself
when it comes to the exact cost of bandwidth, but suffice it to say
that bandwidth is, at least for MMORPGs, a substantial part of the
operational budget. So there is need to limit network bandwidth
usage.

MMORPGs have a substantial need for patching, this is no
secret. WWIIO had a ~70M mandatory patch upon release, updates and
patches of various titles frequently demand similar downloads. To
address this issue, I came up with a design to offload the bandwidth
demands of the game operator. To my knowledge, noone has tried this
solution yet.

Recently peer-to-peer (P2P hereafter) technologies have received a
lot of attention. The gaming industry is interested in control and
I'm not terribly surprised to see that P2P has not been a popular
design feature. In online games, additional problems in the form of
hacking and cheating in various ways add to developers' cautiousness
for the subject. I believe it is doable, and this is what I set out
to do:

  Terms: 

    system - all the services and machines operated by the provider
    of the application

    service - a server providing files, maintained by the owner of
    the system

    client - an application running on a customer's computer

    provider - an application making something available to others,
    either a service provided by the system or a feature of an
    application on an end user's computer)

  - On demand, dynamic background patching. The client receives
  updates of the player's surroundings and then knows what media it
  needs. If that data is missing or corrupt, the necessary files are
  requested from the server.

  - Clients are registered as they connect, collecting up- and
  downstream bitrates and whether the client can support hosting
  media. This is a run-time registration only, not customer or
  subscription which are separate as needed by the system.

  - The server system provides download services and client pairing

  - Upon request for a media file, the download service either
  provides the file directly or notifies one client to act as server
  and the other where to get the file. When assigning P2P client and
  server roles, the service must track current bandwidth usage at
  both ends. The client requesting the download also receives a
  checksum map of the file requested with additional data necessary
  to detect corrupt or tampered data.

  - Clients verify their media files via checksums. If corrupt data
  is encountered, the download service is notified and a new data
  provider assigned. The service may take steps to correct the
  download provider (the client that provided corrupt data - at the
  least, the client should be forced to repatch the data on its own
  installation)

  - To additionally prevent malicious tampering, media files are not
  taken from a client in their entirety but in fragments. Typically
  only a few kbytes in each block. This way only blocks that pass
  checksum tests can be provided, and a checksum of the complete
  file can also alert the system to tampering or corruption.

  - The stream for download must be throttled so as not to interfere
  with actual game updates (patching bitmaps must not prevent the
  user from seeing what is happening in the game)

  - The media stream (the P2P stream mentioned above) is separate
  from the game update stream, ie no game data updates occur in this
  stream.

The main hurdles to overcome to make this implementation a reality
and, most of all, useful are as I see them:

  - Server processing. The workload for keeping the system in
  operation is tremendous. I honestly think the advantages in
  bandwidth costs should pay the hardware expense, if not at this
  time then within a year or two. Continuous operation also pays off
  - buying a server is a one time cost (let's skip other financing
  options for now, they don't change basic facts), while bandwidth
  costs every day, every hour, every minute.

  - Customer client bandwidth. Widespread use of broadband services
  is necessary. Also necessary is that clients be configured to
  accept and allow P2P usage. MMORPGs today all use less than what a
  56k modem provides and most of the bandwidth of broadband
  customers is unused.

  - Clients must either be forced to accept being used as hosts
  (less than tasteful if you ask me) or given a good reason to
  provide it, such as subscription cost reduction or otherwise
  improved service.

To additionally improve the concept, one could 

  - Use multicasting for downloads (this will probably not be
  feasible until IPv6 is widespread (2006?) - correct me if I'm
  wrong - I know very well it's possible with IPv4, but is it
  practical? I think not)

  - Use an engine where all media is seamless. There are no zones,
  the playing area can be continuously expanded on the edges.

  - Require no data (or very limited amounts of data) for
  installation. (The application can be distributed on CDROM or DVD,
  that is just more convenient than downloading)

  - LOD schemes for all media, possibly with a "lowest acceptable
  level" of media file installation which must be present for the
  user to commence playing. Once the client is verified to have the
  necessary basic files installed, the rest can be left to
  background patching.

  - Any type of data can be patched this way - this applies to for
  example terrain, textures, models, animations, audio clips or
  binary excutables and whatever other data the application uses.

  - The client installation could maintain the data in separate
  files or in indexed composite files. This is not relevant to the
  construct, just a separate improvement.

  - A verification system must be in place on the client. The amount
  of checksums etc that must be transmitted to verify a client's
  data must be limited somehow - this is a bigger problem than it
  might at first seem, if one takes the unknown state of the client
  machine into account. (This is probably where the least acceptable
  client installation comes in.)

Notes:

  - The concept of an in-application P2P system (imagine Windows
  Update having a built-in Napster function to offload Microsoft's
  servers when propagating patches - Kazaa, a file sharing service
  like Napster, uses its own service to distribute its binary btw,
  I'm sure there are more examples) is certainly applicable in other
  contexts and the demands are quite similar. I merely choose this
  (online games) context because that was the application it was
  designed for, plus it's one of the best fits for the scheme that I
  can think of. There are security concerns, certainly. Such
  concerns is the reason we study computer security and crytography
  and so on - a problem requires a solution, not resignation.

  - The 2006 date is what I recall to be the deadline for when the
  world needs to have IPv6 in place, as well as the date for when it
  could reasonably be in place. Happy coincidence, that.  - As a
  reference to a block-comparing file synchronizing service, see the
  implementation of rsynch, a very nifty *nix utility/service. (I
  have not gotten to this particular point in the implementation
  yet, I was only told that it does things this way - it is relevant
  for block-wise synchronizing, not for P2P which it does not do).

  - The 3D patent mentioned - I don't know whose it was or when it
  was published. I know it was reported on Gamasutra among others, a
  search in their archives might be in place.

  - I am not unaware of the undeniable insecure nature of a client
  installation. Any data that is present on a client machine (in
  persistent or temporary memory) must be considered cracked and
  tampered with.

  - Run time properties of the scheme - the scheme requires a
  massive scenario to be worthwhile. It also becomes profitable only
  in cases where there are substantial downloads.

  - The mandatory question: where is my download so I can use this? 
  Answer: Nowhere. I have no working implementation of this,
  especially not in its entirety and in particular, nothing for
  free.

Perhaps some on this list can find the idea useful or interesting,
perhaps you can spot a fundamental flaw. I realize that it applies
very little to MUDs, but it certainly applies to MMO and therefore I
submit it here. If this topic has been discussed before on the
MUDdev list or elsewhere, I am not aware of it (yes, I have
searched).  Any additional data on bandwidth usage in existing
MMORPGs would be interesting to know as well. The bandwidth
requirements of the client is usually well known (as stated on the
box) but the actual usage and/or cost for the game operator would be
interesting to know. I recall a statement from Verant saying the
bandwidth cost was somewhere in the range of 35-50% of their cost of
operations - don't know where I saw it, sorry (I have tried to find
it again but failed).

Please let me know of any reactions. Apologies for the somewhat
pompous format - that's just my style I guess. I'm not trying to
stake claims for fame. Apologies also for the longish post, I hope
some of you found it readable.

- Olof

/em fully expects to be told this is a silly idea, still wants to try
_______________________________________________
MUD-Dev mailing list
MUD-Dev at kanga.nu
https://www.kanga.nu/lists/listinfo/mud-dev