[DGD]Loading a file into an array

[Par Winzell]

Risto,

 > I have this little problem: I should load an over 64kb (which is the maximum
 > size for strings in DGD) file into the memory somehow. When trying to read
 > the whole file into a string, I just get an error message, "String too long"
 > When I try to read the file into an array line by line, I get 
 > "Array too large".

DGD has a number of fixed and floating limitations. The 64K length limit
on strings is fixed in the source. The maximum array size is settable in
the configuration file. While occasionally irritating, these limits are
really good ideas.

One reason is that there are plenty of O(N) array/string functionality
tied to simple operators, and it would be tacky of the driver to freeze
for 3 seconds simply because you're attempting to modify the 19th byte
of a 7-megabyte string (which would prompt a copy), or add "foo" to the
beginning of it.

Worse, code of the type,

  str = "-";
  for (i = 0; i < N; i ++) {
    str += "courtney-";
  }

which looks very innocent is actually O(N^2) and thus deserves an upper
limit.

 > So, anyone have any ideas (or a working algorithm) to do this?

The thing to do is to write e.g.

	/lib/bigstring.c

where you could work e.g. on an array of strings. You lose the operator
convenience but that may be just as well for such potentially expensive
operations...

I've found in the past that the single most important trick to scaling
up to thousands of users is to eliminate every single O(N^2) bottleneck
which means using either mappings instead of arrays, or writing a fast
array (or string) addition implementation in LPC which keeps a "current
write index" variable to remember where it's writing, and only extends
the array/string in large chunks. For example,

private mixed *buffer;
private int    ix;

private void arr_new() {
  arr = allocate(8);
  ix = 0;
}

private void arr_append(mixed *arr) {
  int i;

  while (ix + sizeof(arr) > sizeof(buffer)) {
    buffer += allocate(sizeof(buffer)/2);
  }
  for (i = 0; i < sizeof(arr); i ++) {
    buffer[ix ++] = arr[i];
  }
}



This is a very simple example, and in fact I have not compiled it, so
treat it as basic idea only. My hope is that this kind of 'tail buffer'
will make its way into the driver one day, rendering O(N^2) string and
array operations O(N) at some cost in space.

Pär

List config page:  http://list.imaginary.com/mailman/listinfo/dgd