[MUD-Dev] Persistant storage.... My current idea.

[Ben Greear]

On Wed, 1 Apr 1998, J C Lawrence wrote:

> A very simple cheat which performs very nicely is to do something as
> follows:
> 
>   Every record in the DB is identified by a unique record # of a
> signed integer type.
> 
>   The DB consists of two files, the database itself, and an index.
> 
>   The index file is an array of the following structures:
> 
>     struct {
>       off_t record_pos;   // Offset of the record in the DB
>       size_t record_len;  // Length of the record in the DB
>     }

I still see no reason to store the index on disk.  It just means another
write everytime I modify the DB significantly.  My hash table will be
exactly that however, just stored in RAM instead of disk.

> 
>   The N'th structure in the index file, located by seeking to an
> offset of (N *sizeof (struct)) and then reading sizeof (struct) bytes, 
> holds the data for the record in the DB with a record number of N.

The object id numbers will not be synchronus, indeed, they will vary
wildly as the high bits will be the server id, and the low bits will be
the object id on that server.  I'll probably use longs or two integers,
eight bytes at any rate.

> 
>   Seeking to offset record_pos in the DB file and reading record_len
> bytes will get you the wrapped contents of that record.
> 
>   The DB file is an amorphous binary blob of no particular pattern.
> It contains records of variable length which follow a particular
> pattern:

I'm using java, I think my "JstoreHeader" will be something like this:


long prev_obj_seek_posn;
long next_obj_seek_posn;
String cur_obj_class_name; //ie, get this class, it will know how to
                           //read itself.
int max_length; //of this object's allocated file space
int cur_length; // how much it's using now...although this might not
                // be too useful.
Not sure whether or not I'll try to wrap the contents in the header, at
least explicitly.  Don't think so because at boot time, I'll just read
all headers to make the hash table, skip (seek) right over the data
portions.

>   There is reason behind the signatures and the like.  Look at tdbm
> and YOODA for working models.  The base reason here is so that the
> index file can be reconstructed from the DB file, and so that gross
> format corruption can be detected.

Yes, this is highly desireable and I will definately have it.

> 
>   New records are added into either the first free space in the DB
> file that is large enough to hold them, or some "best fit" equivalent.
> As old records are deleted this opens "spaces" that new records (with
> potentially very different record numbers) will be inserted into.
> Order has no importance in the DB file -- that is what the index file
> is for.  

I'm just going to put them on the end.  I'd wrather waste disk space
than take the time to be clever.  Besides, as the objects' images on
the disk will change, I'll need some room to grow and shrink w/out
affecting the neighbors.

> I now have a dedicated 'net connection at home.  In celebration I'm
> moving the list to Petidomo (see www.petidomo.com), and adding
> searchable web archives of past list traffic (yes, email addresses
> will be munged), and a backing FTP site.  This transfer was supposed
> to be happening today.  It didn't and it won't, not today.  Sorry.
> RSN.

Damn, I'm jeleous...gotta give Cox and US west a swift kick in the
arse..saw 2 MBS connection (wireless T1) for $360 a month..but it would
have to make money to be attractive to me....  Just wait for DSL or
the cable modems..whichever get here first :)

Ben Greear (greear at cyberhighway.net)  http://www.primenet.com/~greear 
Author of ScryMUD:  mud.primenet.com 4444
http://www.primenet.com/~greear/ScryMUD/scry.html