Trying to allocate too much memory is not handled gracefully

[Peter Bruin <P.J.Bruin@math.leidenuniv.nl>]

Bonjour,

When one tries to allocate an impossible amount of memory using
allocatemem() or default(parisizemax), GP does not always handle this
gracefully.

1) allocatemem(s) causes GP to crash for values of s >= 2^63 (on a
   64-bit system, with parisizemax set to 0):

gp> allocatemem(2^63)
  ***   Warning: not enough memory, new stack 4611686018427387904
  ***   Warning: not enough memory, new stack 2305843009213693952
  .........
  ***   Warning: not enough memory, new stack 140737488355328
  ***   Warning: not enough memory, new stack 70368744177664
  ***   Warning: new stack size = 9223372036854775808 (8796093022208.000 Mbytes).
Segmentation fault

This is due to a "minss" that should be a "minuu".  While debugging
this, I noticed in addition that when the requested size s needs to be
halved, it may no longer be aligned as it should.  Finally, when s is
extremely close to 2^64, it is not halved but set to the minimum
possible stack size due to an overflow.

2) default(parisizemax, s) prints the requested value s instead of the
   value that was actually allocated:

gp> default(parisizemax,2^63-1)  \\ maximum value of this default
  ***   Warning: not enough memory, new stack 4611686018427387904
  ***   Warning: not enough memory, new stack 2305843009213693952
  .........
  ***   Warning: not enough memory, new stack 140737488355328
  ***   Warning: not enough memory, new stack 70368744177664
  ***   Warning: new maximum stack size = 9223372036854775807 (8796093022208.000 Mbytes).

The attached patch is one possible way to fix these bugs.  It fixes the
output of default(parisizemax), and it seems to make allocatemem(s) work
as expected for all s <= 2^64 - 1.

A somewhat related problem is that allocatemem(s) causes GP to crash
when s is very small, but I have not looked into this.

Thanks,

Peter


diff --git a/src/language/init.c b/src/language/init.c
index d85a58e..8ab4210 100644
--- a/src/language/init.c
+++ b/src/language/init.c
@@ -650,7 +650,7 @@ fix_size(size_t a)
 {
   size_t ps = PARI_STACK_ALIGN;
   size_t b = a & ~(ps - 1); /* Align */
-  if (b < a) b += ps;
+  if (b < a && b < ~(ps - 1)) b += ps;
   if (b < MIN_STACK) b = MIN_STACK;
   return b;
 }
@@ -659,15 +659,16 @@ static void
 pari_mainstack_alloc(struct pari_mainstack *st, size_t rsize, size_t vsize)
 {
   size_t sizemax = vsize ? vsize: rsize, s = fix_size(sizemax);
-  for (;; s>>=1)
+  for (;;)
   {
-    if (s < MIN_STACK) pari_err(e_MEM); /* no way out. Die */
     st->vbot = (pari_sp)pari_mainstack_malloc(s);
     if (st->vbot) break;
-    pari_warn(warnstack, s>>1);
+    if (s == MIN_STACK) pari_err(e_MEM); /* no way out. Die */
+    s = fix_size(s >> 1);
+    pari_warn(warnstack, s);
   }
   st->vsize = vsize ? s: 0;
-  st->rsize = minss(rsize, s);
+  st->rsize = minuu(rsize, s);
   st->size = st->rsize;
   st->top = st->vbot+s;
   st->bot = st->top - st->size;
@@ -721,7 +722,8 @@ parivstack_resize(ulong newsize)
   if (newsize == pari_mainstack->vsize) return;
   evalstate_reset();
   paristack_setsize(pari_mainstack->rsize, newsize);
-  s = newsize ? newsize : pari_mainstack->rsize;
+  s = pari_mainstack->vsize;
+  if (!s) s = pari_mainstack->rsize;
   pari_warn(warner,"new maximum stack size = %lu (%.3f Mbytes)", s, s/1048576.);
   pari_init_errcatch();
   cb_pari_err_recover(-1);