[MUD-Dev] TECH: ColdStore and MUDs

[ryan daum]

Hi, I've written the start of a simple MOOish thing in C++ using
ColdStore (see below) ... If anybody is interested, drop by
moo.xao.com 7777 where we discuss it.  Sourcecode for ColdStore (and
the mud) is available by CVS from

  http://sourceforge.net/projects/coldstore

What is ColdStore?

ColdStore is a toolkit for developing transparently persistent
applications.

While all applications are about data -- information, whether to
manipulate it, calculate it or display it -- most applications also
have some need to persist data, to keep it around over the
long-term.  Transparent persistence is about letting the program use
the same data structures it uses in-memory to persist the data
long-term.

ColdStore is extremely fast.  ColdStore can make any piece of
allocated memory persist.  What the programmer can create in memory,
ColdStore can map to disk.  ColdStore can persist data of arbitrary
complexity; everything from linked lists to complex tree
structures. ColdStore provides all of this (and more) with no extra
cost in development time.

ColdStore is written in C++, and can easily be integrated with a
wide variety of other languages - it currently has early support for
Python (via a python module), an XML parser, a forth-like language
(Chaos) and a C-like object-oriented prototyping language (Freon).

ColdStore is implemented with several cleanly separated layers;
programmers can write code to any layer, using higher-level layers
for ease of implementation. while retaining close integration with
the underlying store, or using lower-level layers for direct access
to the store without using the facilities of the higher levels.

The key portions of ColdStore are listed below, along with their
associated features:

  "Layer0"

    * A framework for doing ultra-fast persistence for programs.

      We use the mmap call to map memory directly to disk, allowing
      a process's address space to directly map to a disk file.
      This allows any allocated portion of memory to be immediately
      available to the program upon restart.  This approach allows
      for an almost zero per-object load and save time for objects,
      and is almost totally transparent to the programmer. In
      addition, memory portions are allocated in extants and
      neighbourhoods, allowing for optimization of disk access based
      on the type of data involved.

  "Layer0.5"

    * A toolkit for the interning and management of data,
      procedures, objects, etc. that are held in dynamically
      loadable libraries (i.e. ".so" files)

  "Layer1"

    * A layer of fast, flexible (but simple) data types for C++
      applications.

      Our type library includes Integers, Strings, regular
      expressions, Tuples, Lists, dictionaries, Namespaces, etc.
      These classes have been optimized for use with the persistence
      layer (layer0 as we call it) and take full advantage of the
      ability to allocate by extant.  Our type layer is loosely
      based on the Python language's type model, but with many
      improvements.

    * A simple Forth-like language (called "chaos") that can be used
      for testing or scripting simple applications that use the type
      library.

  "Layer2"
 
    * A set of classes for using ColdStore to build programming
      language interpreters on-top of the above layers.

      These classes allow interpreter writers to worry about details
      of syntax and structure without having to worry about arcane
      implementation details related to virtual machines.  Languages
      implemented using our infrastructure gain from automatic
      persistence, garbage collection, our extensive type library,
      and a library of already existing syntax tree node
      implementations.  Rather than compiling from syntax to
      bytecode like many virtual machines (such as Java, Python,
      etc.), we store and interpret programs directly as their
      abstract syntax tree (AST) format, shaving an extra cycle off
      the interpretation cycle.

  "Toolkit"

    * A toolkit for writing prototype inheritance object-oriented
      applications.

      Most object-oriented languages lay down a strict distinction
      between classes and their instances.  A class roughly defines
      a structure and its behaviours while the instance uses them.
      Prototype inheritance collapses this distinction: all objects
      are potential classes.  Any object can have children
      (instances), and can thus be treated as a class in and of
      itself.  Prototype based systems (like Self, NewtonScript,
      JavaScript, etc.) allow for extremely rapid development cycles
      and, when combined with some sort of multi-user access to the
      object database, are ideal for large multi-programmer
      projects.  For more information on prototype inheritance, take
      a look at:

        http://www.mactech.com/articles/frameworks/7_6/Prototype-based_OOLs_Evins.html

      Our toolkit includes a prototype-based Object class, along
      with various convenient extra type utilities, and a set of
      extensible example root prototypes to implement very commonly
      used features.

  "Freon"

    * An object oriented prototyping language (called "Freon").

      Freon combines a C-like syntax with a clean object oriented
      syntax, exception handling, garbage collection, and a simple
      form of metaclassing.  While whole projects could be written
      in Freon, it is ideally suited for prototyping objects that
      can be later easily rewritten (method by method) using native,
      compiled C++ methods.  Freon implements very strict
      encapsulation and data hiding, making it possible to write
      quite secure applications, with multiple programmers
      collaborating on large projects, extending objects in
      real-time, without stepping on each other's toes.

  Misc.
 
    * An XML parser.

      Storage of complex tree structured data, like XML, is very
      awkward using existing relational databases.  At the same
      time, parsing a large XML document every time a program is
      started adds significantly to its execution and development
      time. ColdStore is ideal for storing and manipulating data
      that is stored in XML-based formats because once the data is
      first parsed into its internal tree data structure, it is
      ready for use with no extra cost.

      In addition, because the internal tree structure we use to
      store XML data is based on the same prototype Object model we
      use for developing other persistent applications, it is
      possible to easily attach behaviours and methods to your XML
      elements (using freon, C++, C, or any languaged developed in
      layer2).

    * A GUI (GTK+-based) object browser for browsing objects and
      methods defined in a ColdStore session.

    * A metaclass for use in C++ applications which make programming
      for the toolkit much simpler and intuitive for C++ coders.

      OpenC++ (http://www.csg.is.titech.ac.jp/~chiba/openc++.html)
      is used as a preprocessor for the ColdStore C++ source code.
      It allows us to add extra features to C++ classes at compile
      time that would ordinarily be difficult to express.  We use
      OpenC++'s metaclasses to add methods for accessing ColdStore's
      prototype objects, namespaces, and methods in a very intuitive
      fashion.  C++ programmers can set members, define methods, and
      send messages to ColdStore toolkit objects in a manner that is
      almost identical to writing pure C++ classes itself.

    * A module for accessing any ColdStore object or data from
      Python.

      Python programmers can take full advantage of all of the
      ColdStore layers and toolkits from their programs.  Although
      we cannot provide transparent persistence for Python's own
      native types (yet), most Python types can be easily converted
      to their ColdStore equivalent and transparently used.
      ColdStore persistence is orders of magnitude faster (for large
      data sets) than Python's own "pickle" and "shelve" modules.

____ ryan daum ____________________________________ ryan at techno.ca _________

_______________________________________________
MUD-Dev mailing list
MUD-Dev at kanga.nu
https://www.kanga.nu/lists/listinfo/mud-dev