[DGD] Hydra

[Felix A. Croes]

Today I released Hydra version 1.3.12.

I have been working on DGD since 1992, and on Hydra since 2002.  Although
DGD muds peaked around 2005, I think there is still a case for LPC as a
programming language, for error handling with atomic functions, for the
event-driven model (it has been fun to watch the node.js folks rediscover
old LPmud wisdom), and for object-oriented database management systems.
So I continued developing DGD as a hobby after it stopped being my day
job.

Hydra was born from the realisation that DGD's event-driven tasks
could be parallelized: let multiple tasks run concurrently on copies
of objects, and when they finish, commit the changes when the objects
have not been modified by any other task in the meanwhile; otherwise,
lob off a Hydra's head (discard the task's changes) and let it grow
back (reschedule the task to try again).

What makes this especially interesting is that to the LPC developer, the
concurrent execution of tasks is invisible, and it appears that tasks
are still run in sequence because the commits are serial.  The complex
machinery of parallelization is hidden from view.

Of course, the machinery of parallelization was not hidden from my view.
Developing Hydra has been a very time-consuming hobby.  For the past two
years, I have been working on optimizing the locking architecture,
culminating in today's release.  The multiple CPU, multi-core future that
I expected in 2002 has definitely arrived.  Hydra is well suited to take
advantage of the modern multi-core CPU architecture.

Not on all CPUs, however.  As far as Hydra is concerned, hyper-threading
is a mis-feature.  Any type of feature sharing between hardware threads
or cores is likely detrimental.  Other features, such as "turbo boost"
(increasing single-core performance when no other cores are in use)
complicate measuring the benefits of multi-core execution accurately.
Furthermore, Hydra has been hard hit by this year's speculative execution
bugs.  The mitigations for Meltdown, Spectre et al slow it down
considerably.

Additionally, Hydra does not perform equally well on all operating systems.
I have been using macOS as my main development system; unfortunately,
Hydra scales abysmally on macOS.  I have run some tests on a 2008-era dual
CPU system with a total of 8 cores, which thankfully lacks hyper-threading
and a host of other features that modern CPUs are often blessed with.  I
see that Hydra scales well on Windows 10 and Solaris 10, but when using 4
or more threads dedicated to running LPC tasks, Linux is slightly faster.
Interestingly, with fewer than 4 threads dedicated to LPC tasks, Linux is
much faster than other operating systems I tested.  Overall, I see best
performance on Linux kernels without speculative execution bug mitigations.

Since testing on a 10 year old system is hardly definitive, I want to ask
for volunteers to test Hydra on such modern systems as they have available.
The test should be run on bare hardware.  Running it virtualized slows down
Hydra a lot, and is also likely to have a large impact on other VMs running
on the same hardware.

The test itself is simple.  Download Hydra from

    http://download.dworkin.nl/hydra/

and the test from

    http://download.dworkin.nl/hydra/benchmarks/bm002.zip

Edit test.dgd from bm002 to let the base directory point to wherever you
put bm002/src, and run Hydra specifying the number of threads dedicated
to LPC tasks, for example:

    hydra-1.3.12 -1 test.dgd > test-1
    hydra-1.3.12 -2 test.dgd > test-2
    hydra-1.3.12 -4 test.dgd > test-4

Then post the results, plus CPU information (/proc/cpuinfo on Linux) to
the mailing list.

Test results for more than 4 cores are especially interesting.  Hydra runs
more threads internally than are used for LPC, and you should leave some
room for the other OS tasks, so tests up to the number of cores minus two
are meaningful.  If your system has hyperthreading, divide the number of
"cores" reported in /proc/cpuinfo by Linux by 2.

The benchmark is synthetic and does not test Hydra's true performance;
rather, it attempts to measure the performance of Hydra's locking
architecture.

Regards,
Felix Croes