[DGD] Rewriting the main loop

[bart at wotf.org]

On Sat, 1 Dec 2018 17:27:20 -0800, Raymond Jennings wrote
> As for the reason:
> 
> It is my personal opinion that DGD should be as responsive as 
> possible, and it should place human interaction before internal 
> timing from callouts.

How so?

call_outs can be very useful as part of that interaction, unless everything a
user could do can be 'answered' with very little work.

> 
> This implies pushing background work like swapping and garbage collection
> to the, well, background.

Swapping is essential for keeping things responsive, and at some point,
keeping things working at all.

For example, my test instance of lpcdb is currently down, why? It is running
out of memory because of a 'table scan' (actually, a column scan, as table
scans are broken up in independent tasks for each column) does not give dgd a
chance to swap out objects that already have been scanned.

This is an issue in my lpc code for which I've been creating a framework that
allows for easy breaking up such tasks in continuations, but until that time,
I either need to reduce the amount of data in this specific table, or give DGD
more memory, neither of which are a realistic option in this case.

Bottomline is, call_outs and swapout have a role to play in user interaction,
and I don't think you can say that those could just wait for 'idle time'. Now
there certainly are cases where they could, but those do imo not negate the
cases where they can't.

Bart.
--
https://www.bartsplace.net/
https://wotf.org/
https://www.flickr.com/photos/mrobjective/