[DGD] sig11, snapshot restore and stack size

[bart at wotf.org]

You could stick 16384 strings with a few k  
random data each in an array to produce a 
multi megabyte object. The exact case I had 
isn't easily reduced to an obvious example, 
it is part of a full text search engine and 
is not obvious without the other components 
of that engine.

Distributing the data over multiple objects 
is a valid approach in the case I have and 
was already in the plan. Multi megabute 
objects don't play very nice with swapping 
in general and while splitting up the data 
gives some more overhead, it also performs 
in a much more predictable and consistent 
way.

Bart

On Wed, 16 Mar 2016 12:42:48 -0700, Raymond 
Jennings wrote
> On Wed, Mar 16, 2016 at 11:05 AM, 
<bart at wotf.org> wrote:
> 
> > I have a rather huge object in a 
snapshot file, and dgd was ending in a
> > sig11
> > on actually trying to use said object 
after a restore.
> >
> > Looking at what was happening made me 
end up in d_conv() where a buffer
> > (buf)
> > is allocated with ALLOCA.
> >
> > It tried to allocate approx 14mb, while 
the maximum stack size was 8mb.
> > According to man alloca the resulting 
behavior is undefined.
> >
> 
> After changing the stack size to 32mb, the 
object could be restored
> > properly.
> >
> > However, this prevents the issue in this 
specific case, but the potential
> > of
> > overflowing the stack due to such an 
object is still there, and its not
> > obvious until after you try an actual 
restore. You'll have to look with a
> > debugger at how large your stack should 
be to avoid a particular case.
> >
> > I probably shouldn't be creating such a 
large object, but I also don't
> > think
> > just assuming it will fit the stack is 
such a good idea, actually, I'm not
> > sure this buffer should be on the stack 
at all considering the rather
> > unpredictable size it can have.
> >
> 
> Anything that can be arbitrarily large 
should indeed most likely be
> allocated on the heap rather than on the 
stack.
> 
> I think dworkin actually made a similiar 
change for allocating memory
> during directory listing, which is much 
smaller than multi megabyte.
> 
> If you could narrow down the code in 
question or cook up a test case 
> it would probably qualify for provoking a 
bugfix.
> 
> Any thoughts?
> >
> 
> My advice is to cook up a test case and 
send it to dworkin for
> patching...and in the meantime, yes, 
objects that require restores of
> multiple megabytes are almost certainly 
bad design and you should 
> try to break the object up into manageable 
chunks.
> 
> > Bart.
> > --
> > 
http://www.flickr.com/photos/mrobjective/
> > http://www.om-d.org/
> >
> > 
____________________________________________
> > 
https://mail.dworkin.nl/mailman/listinfo/dgd
> 
____________________________________________
> 
https://mail.dworkin.nl/mailman/listinfo/dgd


--
http://www.flickr.com/photos/mrobjective/
http://www.om-d.org/