[DGD] DGD and *dbm

[bart at wotf.org]

On Mon, 21 Jul 2008 14:18:33 +0200, Felix A. Croes wrote
> bart at wotf.org wrote:
> 
> > The idea I'm playing with is to add support for one of the *dbm packages (bdb,
> > gdbm, tdbm etc). This would serve a number of purposes, including storage of
> > huge amounts of (potentially large) key:value pairs, and possibly, use it to
> > share data with external applications.
> >
> > The questions I have:
> > Because any such dbm might take a fair amount of time to return some results,
> > I'm considering a callback mechanism to return results. This breaks
> > continuity, ie, you have to request data, and then wait for the callback
> > instead of being able to do something like var = dbm_get(key). I'm not sure if
> > this is really needed when requesting a single key:value pair tho. Any
> > experience with this?
> 
> I have even more questions:
> 
> Will C-level calls to the database be handled by DGD's interpreter,
> or will the database run in a different process or thread?

Initially, by DGD's interpreter, possibly as a seperate process to prevent it
from blocking the interpreter while the db is busy. 

If this proves unworkable (I could see some reasons for this), it will be
easier to move the entire thing into a seperate process and use a connection
to communicate with it. That is however the thing I'd like to avoid (it is
probably desirable for larger, more serious setups, but having it integrated
into the driver is a lot easier for my 'hobbymud' setup. It would also be a
lot 'cleaner' in that it wouldn't touch any DGD internals I imagine, and yes,
that is the networking discussion all over again)

> 
> Will the database play fair with atomic functions?  How will an 
> atomic rollback affect it?

At least bdb and tdbm support a form of transactions, a commit could happen
after DGD completed the atomic execution with no rollback. I haven't looked at
this too closely yet, it is an important consideration however. As a worst
case workaround, enforce that the dbm related kfuns can't be used from within
atomic code (not feasable for DGD/MP I suppose)

> 
> Do you intend it to work with DGD/MP in the future, where even non-
> atomic code can be rolled back?

I'd love to, but for now I can only target what I can actually work with :)
In all seriousness, I don't expect such a thing to directly work on DGD/MP
from all the things you have explained about it so far. Support for DGD/MP
will be similar to the network package.. Not at all right now, but once I can
get my hands dirty with MP, I'd like to see if I can port the code to it. I
expect this to be a fair amount of work also.

> 
> > Since there can be more key:value pairs in such a dbm then fit in an array or
> > mapping, it won't be possible to get a list of all keys or values directly, I
> > am considering returning a range (of max max_array elements), based on
> > start/end arguments (ie, arr = dbm_keys(x,y); ). Any thoughts on this?
> 
> Have you considered arrays of arrays (of arrays...) of results?

Yes. I'm not sure which way would be more 'convenient' to LPC coders, its
definitely an option, it would just take a bit longer to produce the result.

> 
> What sort of information are you allowing to be stored in the database?
> Strings only, everything but objects, or anything whatsoever?

Everything but objects (Everything encoded in the format used by save_object)

Bart.
--
Created with Open WebMail at http://www.bartsplace.net/
Read my weblog at http://soapbox.bartsplace.net/