[MUD-Dev] TECH: DBM vs BDB speeds (was: AmigaMud DB questions)

[Bruce]

Jon Lambert wrote:

> The only work I know of that has looked at optimizing massive
> insertions in particular has been TDBM and dxStore.  I believe
> similar problems prompted the creation of both.  Neither is quite as
> mature as GDBM or BDB (meaning possibly buggy).

Hmmm, I had forgotten about dxStore. :) TDBM looks like it is well
past the point of being inactive into looking dead.  A shame.

> I created a special object called $catalog.  It's sole function is
> maintaining those global symbols.  And by globals I mean names of
> objects.  Anyways since its just another object it lives in cache
> with the rest of objects.  And the high access frequency tends to
> keep it there.  I suppose it might be a useful concept for Cold as
> it would necessarily remove any objname->objnum index records, and
> make the in memory hashtable unecessary.  Of course in your
> particular case, that would probably be one massive object.  :-)

Mmmmmm. Way too massive. :)

I don't mind the objname->objnum index records.  And we still have to
have the objnum->(size,offset) index, so it isn't a lot of extra work
to maintain both.

> I guess maybe if the problems of why it's necessary to do a
> decompile and recompile are resolved when one does a driver upgrade
> then it wouldn't be that big of an issue.  I assume that data
> changes in some dramatic way between driver updates?  It shouldn't.
> Which is to say that I don't think it should at this late stage in
> development.  And if it did some sort of automatic upgrading would
> be more useful.

Well, usually, you can get away without that decompile/recompile.
Just that there are some dependencies within the bytecode values on
the driver configuration, the set of ops/functions, and the set of
native methods.  In development over the last couple of months, we've
added new ops/functions as well as new native methods.  Furthermore,
most of the serialization code was rewritten, including some changes
in the on-disk format to make things more efficient.  Those types of
changes necessitate a re-creation of the binary database.
Furthermore, there was -no- reason for it to take as long as it was,
and is usually a quick and painless process.  It just wasn't scaling
well.  I'm confident that with some work, we could get the current 2
hour time for a 1.5G DB down to under an hour, but that's not time
that I have at the moment.

  - Bruce

_______________________________________________
MUD-Dev mailing list
MUD-Dev at kanga.nu
https://www.kanga.nu/lists/listinfo/mud-dev