[DGD] Code libraries for DGD and Hydra

[bart at wotf.org]

On Wed, 19 Sep 2018 06:13:15 +0200, Felix A. Croes wrote
> When writing code for Hydra, one has to keep one thing in mind: the
> current task may fail to commit, and have to be rescheduled, if one
> or more of the objects accessed by the current task were modified by
> another task, running concurrently.
> 
> This can be prevented by keeping the number of objects accessed by
> the current task low.  Tasks that access many objects can be broken
> up into smaller subtasks, chained together by callouts.

Those are 2 things I've been doing a lot, and actually the way lpcdb stores
its data is intended to prevent such collisions, primarily by trying to have
data which has a high chance of being accessed from multiple tasks in separate
objects. Its also one of the motivations behind not using btree like
structures where lookups have to go through the same root node(s).

<snip> 

> As a special case, consider an object which is central in some way,
> and which has to be updated often by a variety of tasks.  Normally,
> such updates would run a high risk of causing the task to fail to
> commit.  But when call_out_other() is used, the calls are indirect,
> and the central object can process the update from the
> callout-started task.  Hydra can efficiently manage a large number of
> pending callouts per object.
> 
> Call_out_other() can also be used to implement the map/reduce
> coding pattern.  One object chops up a large dataset, adds callouts
> with chunks of the data to a large number of other objects, those
> objects process the chunk of data within the callout-started task,
> and then report back to the originating object using another callout.
> The originating object aggregates the results step by step, until all
> the results are in.

lpcdb does this a lot, and this is also what allows it to distribute a
workload over multiple instances. 

Shameless plug for some seemingly little used extension I use a lot.. the
closures extension would provide a clean way to let the called out worker
object do the call back for submitting its result in a more easily secured way
(as it lets you pass a static callback function)... (this and variations on it
are actually the only real reason why I am using that extension)

> 
> There is one caveat.  Naturally, using status(obj) counts as
> accessing the objects state, and therefore it is one of the things
> that call_out_other() is not doing.  But status(obj) includes
> information about callouts, and querying this state will explicitly
> cause a collision with any changes to those callouts made by other
> tasks.  Thus, status(obj) and status(obj)[O_CALLOUTS] become very
> expensive operations.

That is good to know, and means I'll have to change my call_out handling
somewhat to prevent needing the call_out info from status(obj) in most cases.

Bart.
--
https://www.bartsplace.net/
https://wotf.org/
https://www.flickr.com/photos/mrobjective/