Bug#1057344: libgmp10: major formatted output function bug with %c and the value 0

[Vincent Lefevre]

Package: libgmp10
Version: 2:6.2.1+dfsg1-1.1
Severity: grave
Tags: security upstream
Justification: user security hole
Forwarded: https://gmplib.org/list-archives/gmp-bugs/2023-December/005420.html
X-Debbugs-Cc: Debian Security Team <team at security.debian.org>

I've reported the following bug upstream. Debian/stable is affected
(at least on the testcase below, but the various issues are probably
related).

With GMP 6.3.0, the formatted output functions do not handle %c
with the value 0 correctly. For gmp_sprintf, the return value is
incorrect. For gmp_asprintf and gmp_vasprintf, this is either a
buffer overflow (according to the GMP manual: "The block will be
the size of the string and null-terminator.") or, in case this
is an error in the GMP manual, possible memory corruption when
freeing the allocated memory, if the custom memory allocation
function cares about the size parameter.

Testcase for gmp_sprintf:

------------------------------------------------------------
#include <stdio.h>
#include <gmp.h>

static void test (int flag)
{
  char s[3] = { 1, 1, 1 };
  int r;

  r = (flag ? sprintf : gmp_sprintf) (s, "%c", 0);
  printf ("%4s: r = %d, s = { %d %d %d }\n",
          flag ? "libc" : "gmp", r, s[0], s[1], s[2]);
}

int main (void)
{
  test (0);
  test (1);
  return 0;
}
------------------------------------------------------------

which currently gives:

 gmp: r = 0, s = { 0 0 1 }
libc: r = 1, s = { 0 0 1 }

MPFR has various issues concerning %c with the value 0, but an
attempt to fix them fails due to

  length = gmp_vasprintf (...);
[...]
  mpfr_free_str (s);

which is similar to GMP's tests/misc/t-printf.c file, which contains

  got_len = gmp_vasprintf (&got, fmt, ap);
[...]
  (*__gmp_free_func) (got, strlen(got)+1);

But replacing

  mpfr_free_str (s);

by

  mpfr_free_func (s, length + 1);

i.e. using the return value length instead of strlen(s), also fails.
I suppose that this is related to the incorrect return value.

-- System Information:
Debian Release: 12.2
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, 'stable-debug'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 6.1.0-13-amd64 (SMP w/1 CPU thread; PREEMPT)
Kernel taint flags: TAINT_WARN
Locale: LANG=POSIX, LC_CTYPE=C.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages libgmp10 depends on:
ii  libc6  2.36-9+deb12u3

libgmp10 recommends no packages.

libgmp10 suggests no packages.

-- no debconf information

-- 
Vincent Lefèvre <vincent at vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)