[DGD] Working with parse_string()

[Felix A. Croes]

bart at wotf.org wrote:

> Heh, re: 2. I ran into that more then once.. because of that, and because of
> only having a single string available for passing error information out of an
> atomic function and a number of similar issues, I see upping the string size
> limit as the only available solution.

Maximum string length is set in dgd/src/config.h, i.e. it is
intentionally configurable.  There is of course a cost for handling
large chunks of memory.

The problem of parsing a string longer than whatever you've set your
maximum string length to is very similar to parsing a string of
unknown length; for example, a string received piece by piece on
an internet connection.  What you do is parse as much of it as
you can, and leave the remainder to be re-parsed in a followup call.

Rather than:

    word = /[a-zA-Z]+/
    Array: '{' OptionalWords '}'
    OptionalWords:
    OptionalWords: Words
    Words: Words ',' word
    Words: word

you use:

    word: /[a-zA-Z]+/
    PartialArray: '{' OptionalWords '}'
    PartialArray: '{' OptionalWords ReparseThis
    PartialArray: OptionalWords ReparseThis
    PartialArray: OptionalWords '}'
    OptionalWords:
    OptionalWords: word ',' OptionalWords
    ReparseThis:
    ReparseThis: word
    ReparseThis: word '}'

and the tokens collected by ReparseThis are concatenated and added
as a prefix to the next chunk to parse.

Regards,
Felix Croes