Bug#863217: libgmp10:amd64: gmp_snprintf tries to allocate 18 EB on long strings

[Vincent Lefevre]

Package: libgmp10
Version: 2:6.1.2+dfsg-1
Severity: normal
Tags: security upstream
Forwarded: https://gmplib.org/list-archives/gmp-bugs/2017-May/004157.html

I've reported the following issue upstream. Since GMP tries to
allocate a huge quantity of memory (18 EB) while only a few GB's
may be necessary at most, this may be a security issue (DoS).

Consider the following program:

#include <stdio.h>
#include <gmp.h>

int main(void)
{
  int r;
  long n = -1;

  r = gmp_snprintf (NULL, 0, "%2147483600s%100s%ln", "", "", &n);
  printf ("%d %ld\n", r, n);
  return 0;
}

On my Debian/unstable x86_64 machine (GMP 6.1.2 provided by the Debian
package), I get after 273 seconds:

GNU MP: Cannot allocate memory (size=18446744071562067968)

Wow! 18 EB! Actually this is close to 2^64. I suspect a conversion
of a negative value to size_t as a consequence of an integer
overflow.

As a comparison, if I just use snprintf (glibc function), I get the
result -1 -1 after 12 seconds (the first value must be negative,
the second one may be regarded as unspecified due to the EOVERFLOW
failure).

-- System Information:
Debian Release: 9.0
  APT prefers unstable-debug
  APT policy: (500, 'unstable-debug'), (500, 'stable-updates'), (500, 'unstable'), (500, 'testing'), (500, 'stable'), (1, 'experimental')
Architecture: amd64
 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.9.0-3-amd64 (SMP w/12 CPU cores)
Locale: LANG=POSIX, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages libgmp10:amd64 depends on:
ii  libc6  2.24-10

libgmp10:amd64 recommends no packages.

libgmp10:amd64 suggests no packages.

-- no debconf information