Perl 5.32.1 RC1

Mon Jan 18 00:31:12 GMT 2021

Today I packaged up 5.32.1~rc1 and have uploaded it to experimental.

I would like to include the relevant changes into bullseye, by
cherry picking the following 27 patches listed at the end of this
email. As usual the changes are minimal and generally meet out
stable release policy criteria, so I am happy to ask for an exception
to the soft freeze (a similar rationale was used in #822336, for a
stable point release). I would only do the upload once the final 5.32.1
release is out, expected on 23rd Jan. (Of course, there is a chance
there will be additional patches to include from the final release).

They include all relevant code changes, excluding platform-specific changes.
User documentation is included, but developer documentation is not.
Module::Corelist changes are also omitted.

Like with previous updates of this sort, I will prepare a "debdelta"
file with the relevant changes.

Comments welcome as always.

Cheers
Dominic

commit fa1bc2e4ea514f81f75551539092e7bf6226e8f5
Author: John Lightsey <john at 04755.net>
Date:   Sat Jan 9 12:16:08 2021 +0000

    Make security warnings in the Safe and Opcode modules more direct.

    The current "no warranty" text warning against the use of Safe or
    Opcode for "security purposes" is somewhat ambiguous. These modules
    are not effective sandboxing mechanisms for evaluating untrusted
    perl code and should not be used in that manner.

    Safe and Opcode are, at best, hardening measures that could be used
    in combination with operating system level sandboxing of the perl
    interpreter.

    (cherry picked from commit c150e7c040454515c5acfb8416f2acdb2c0a2bb4)

commit 4190e73eba88675cc39af83a5c4ac5905e42c7e5
Author: Karl Williamson <khw at cpan.org>
Date:   Wed Nov 25 18:20:28 2020 -0700

    Avoid deadlock with PERL_MEM_LOG

    This fixes GH #18341

    The Perl wrapper for getenv() was changed in 5.32 to allocate memory to
    squirrel safely away the result of the wrapped getenv() call.  It does
    this while in a critical section so as to make sure another thread can't
    interrupt it and destroy it.

    Unfortunately, when Perl is compiled for debugging memory problems and
    has PERL_MEM_LOG enabled, that allocation causes a recursive call to
    getenv() for the purpose of checking an environment variable to see how
    to log that allocation.  And hence it deadlocks trying to enter the
    critical section.

    There are various solutions.  One is to use or emulate a general semaphore
    instead of a binary one.  This is effectively what
    PL_lc_numeric_mutex_depth does for another mutex, and the code for that
    could be used as a template.

    But given that this is an extreme edge case which requires Perl to be
    specially compiled to enable this feature which is used only for
    debugging, a much simpler, if less safe if it were to ever be used in
    production, solution should suffice.  Tony Cook suggested just avoiding
    the wrapper for this particular purpose.

    (cherry picked from commit 0cc28fe31b0d416e9c67ecd18b8f38c5833a455a)

commit bdd419192a8b2ab26be9cf5b4b8d7fe6f66b144c
Author: Tony Cook <tony at develop-help.com>
Date:   Tue Dec 8 14:28:29 2020 +1100

    skip trying to constant fold an incomplete op tree

    This code would try to constant fold an op tree like

      relop
        +- null
        +- constant

    which would underflow the stack, potentially crashing perl.

    This is intended as a quick fix rather than as a complete
    solution.

    Fixes #18380

    (cherry picked from commit 08be3ef7f1190d94279ad0b3e13519ac8dc3b0ec)

commit 9aff587fd2011ad85668efc30ed29e794fdb484e
Author: Leon Timmermans <fawaka at gmail.com>
Date:   Thu Dec 17 21:02:46 2020 +0100

    Correctly set LC_CTYPE back in POSIX/t/time.t

    (cherry picked from commit 1cd5f0d5fa70621757c327e1f6ff04bcb220e0a1)

commit 4493dfd4aef2b89c0516020ea0914ae02dfff863
Author: Leon Timmermans <fawaka at gmail.com>
Date:   Wed Jan 6 17:43:15 2021 +0000

    Make DynaLoader on MacOS check library existence with dlopen

    A number of system libraries no longer exist as actual files, even
    though dlopen will pretend they do, so now we fall back to dlopen if
    a library file can not be found.

    (cherry picked from commit d296ead16762852ec34d173616a271c856711f77)

    (with a manual tweak to the $VERSION since maint is behind blead)

commit 95369ab82a0e3669e1631262db9025407341acac
Author: Tony Cook <tony at develop-help.com>
Date:   Wed Jan 6 11:43:16 2021 +1100

    fix a typo

    (cherry picked from commit 64a5950af609117d2098f7616334aff77df9ed63)

commit b620a9f2859aa21d56e1b18ce0c9d3752abaadb0
Author: Tony Cook <tony at develop-help.com>
Date:   Mon Oct 19 15:57:37 2020 +1100

    \K in lookaround might become permitted in the future

    It was disallowed because:

    - it was breaking things (GH #14638)

    - at the time, no-one had the tuits and knowledge to select non-buggy
      semantics and implement them.

    (cherry picked from commit 55afc78396d86da97ab07c25295b00c9a300f081)

commit 57edf1d256c8267868c4c9c44962dd4721d8c2b5
Author: Tony Cook <tony at develop-help.com>
Date:   Mon Oct 19 15:04:28 2020 +1100

    some extra tests as suggested by hvds

    (cherry picked from commit 3c43dd64639f7fbccb6ff32f0e82d007d2cde710)

commit 536520b4a24d012a8aa0690f044f651300258171
Author: Tony Cook <tony at develop-help.com>
Date:   Mon Sep 14 16:00:28 2020 +1000

    don't croak when the \K follows the lookaround assertion

    this also simplifies the flagging for these assertions, since this
    error is now the only thing using in_lookhead and in_lookbehind they
    can be combined into a single in_lookaround.

    Rather than conditional increment/decrement as we recurse into S_reg
    I simply save the value of in_lookaround and restore it before
    returning.  Some unsuccessful or restart paths don't do the restore,
    but they either result in a croak(), or a restart which reinitialises
    in_lookaround anyway.

    Also added tests to ensure that all the different zero-width assertions
    with content trigger the error.

    (cherry picked from commit 80f44cf4982e395989f886220e05dd2071bb205a)

commit 7925f8dfc09ddebcbfa214fdcd854c52090586f9
Author: Tony Cook <tony at develop-help.com>
Date:   Mon Sep 14 15:51:05 2020 +1000

    run_multiple_progs: allow the EXPECT to contain only OPTIONS

    (cherry picked from commit a2f2a4e3143a7f85d874450cced74b7399fd6977)

commit d2f68b387a1247703a3be515c67907131c97b6b5
Author: Tony Cook <tony at develop-help.com>
Date:   Mon Sep 14 15:49:40 2020 +1000

    run_multiple_progs: allow fatal to be turned off too

    in some cases we want to group a test that shouldn't croak with the
    croak tests, in particular to catch regressions when we've
    introduced a croak.

    (cherry picked from commit dd4888cbd881ebd071701fbfef45812a8773a404)

commit 6e48dfa8fc63a182a49aa7d935956f6d77d00e6b
Author: Nicolas R <atoomic at cpan.org>
Date:   Fri Aug 14 16:16:22 2020 -0500

    die_unwind(): global destruction

    Fix #18063

    During global destruction make sure we preserve
    the string by using mortalcopy.

    This is an update on 8c86f0238ecb5f32c2e7fba36e3edfdb54069068
    change which avoided sv_mortalcopy in favor of sv_2mortal.

    (cherry picked from commit 042abef72d40ab7ff39127e2afae6e34dfc66404)

commit b802abc5f61d1342b480ab2dcad2eb9f8d28ca43
Author: David Mitchell <davem at iabyn.com>
Date:   Tue Aug 11 13:58:51 2020 +0100

    list assign in list context: honour LHS undef

    GH #16685

    In

        @a = ($x, undef, undef) = (1))

    @a should have 3 elements. v5.25.6-79-gb09ed995ad broke this and was
    returning one element.

    The fix is simple: that previous commit made it so that elements were
    pushed back onto the stack only if they weren't immortal, so
    &PL_sv_undef was getting skipped. Make it so they always are.

    (cherry picked from commit 282d9dfeb4cea3c2d0335ba78faa3a9db931f1ec)

commit 1da7e62b5d41e8aff3a324906226a6b22ba7fce9
Author: David Mitchell <davem at iabyn.com>
Date:   Tue Aug 11 11:55:46 2020 +0100

    list assign in list context was over-optimising

    GH #17816

    This code:

        my $x = 1;
        print (($x, undef) = (2 => $x));

    was printing "22" when it should have been printing "21".
    An optimisation skips the 'common values on both sides' test
    when the LHS of an assign only contains a single var; as the example
    above shows, this is not sufficient.

    This was broken by v5.23.1-202-g808ce55782

    This commit fixes it by counting undef's on the LHS towards the var
    count if they don't appear first.

    (cherry picked from commit 5b354d2a8a6fea46c62048464c6722560cb1c907)

commit f53b87f8ed3048945d7c727cffcfb0f925c789e5
Author: Hauke D <haukex at zero-g.net>
Date:   Mon Feb 10 20:24:35 2020 +0100

    Note range op behavior change in docs

    This change documents the previous behavior of the range operator with magic string increment in Perl 5.30 and below - the change was introduced in commit d1bc97feec from GitHub #16770 (RT133695).

    (cherry picked from commit f4941eebb5185b70b980fcfea9f7b02f377f1f70)

commit 7a454aa1cba5a4233fd3a0092563e042fc1a6498
Author: Dagfinn Ilmari Mannsåker <ilmari at ilmari.org>
Date:   Fri Aug 21 22:35:37 2020 +0100

    Add Merijn's new email address to Porting/checkAUTHORS.pl

    (cherry picked from commit c2e6241c83ac9b30fb37984a432bff9801cc2a3a)

commit d828ea66796700f472229e6b25f29348ab425cdb
Author: H.Merijn Brand <perl5 at tux.freedom.nl>
Date:   Tue Sep 22 13:17:11 2020 +0200

    Trap SIGBUS when Configure checks for va_copy (#18148)

    On several systems the attempt to determine if we "need va_copy() or
    similar" results in a SIGBUS instead of a SIGSEGV because the fault
    happens pointing at an io device address.

    While reviewing accounting logs during the OpenBSD system build, bluhm@
    noticed a core dump and deraadt@ tracked it down.

    While researching it was discovered that this is a fairly common case in
    other systems as well.

    Original diff by deraadt@

    (cherry picked from commit bec07c3af574f0f22a318c6273a2c4f29afc1251)

commit 94c83adc02828a66324f366d5cee0636d726ef5d
Author: Leon Timmermans <fawaka at gmail.com>
Date:   Mon Jun 29 23:52:31 2020 +0200

    Make perl compile under minix again

    Minix has a #define for si_signo to _info._signo, despite not having
    a struct siginfo; this caused compilation errors.

    (cherry picked from commit 3555a05cbfc80f94a4fe8ae3789f03db69701ca4)

commit 93116e04a61093a3b50ec875085000aff62a2ca9
Author: Hugo van der Sanden <hv at crypt.org>
Date:   Tue Sep 15 14:02:54 2020 +0100

    [gh18096] assume worst-case for GOSUBs we don't analyse

    During study_chunk, under various conditions we avoid recursing into
    a GOSUB. But we must avoid giving the enclosing scope the idea that
    this GOSUB would match only an empty string, since that could trigger
    wrong optimizations (eg CURLYX => CURLYM in the ticket).

    So we mark the construct as infinite, as in the code branch where we
    _do_ recurse into it.

    (cherry picked from commit f4cd5e29bc15621f2ab8fc5d7de0e68e62d43999)

commit 9e9893534814fa6362ae99273d3afbf1402502f9
Author: Karl Williamson <khw at cpan.org>
Date:   Fri Oct 23 07:54:53 2020 -0600

    Fix GH #17278

    This was an assertion failure in regexec.c under rare circumstances.  A
    reduction of the fuzzed test case is now in pat_advanced.t

    The root cause of this was that the pattern being compiled was encoded in
    UTF-8 and 'use locale' was in effect, equivalent to the /l charset, and
    then the charset was reset inside the pattern, to /d.  But /d in a UTF-8
    patterns is illegal, hence the later assertion failure.

    The solution is to reset instead to /u when the pattern is UTF-8.

    (cherry picked from commit bb58640a409949759318542317e353e2241cc408)

commit e744a81b1cb8f9a3df9a4cd784cb2b74d256441a
Author: John Lightsey <john at 04755.net>
Date:   Fri Aug 28 23:39:18 2020 -0500

    Heap buffer overflow in regex bracket group whitespace handling

    The code for skipping whitespace in regex bracket character groups
    was walking past the end of the regex in some cases.

    (cherry picked from commit 90f66c42e4513ae5d907805fbf28b9967a90d6c5)

commit 13ff09012aabaced8a0a0fab40f3c3db32673922
Author: John Lightsey <john at 04755.net>
Date:   Thu Aug 20 15:53:57 2020 -0500

    Add missing boundary check to grok_infnan

    The grok_infnan() function was walking past the end of the string
    while skipping over trailing '0' characters. This is another
    variation of #17370.

    (cherry picked from commit bbd8607595f9856d6e75ed63130034cf645feb4a)

commit 98feab8f7ad1d47a3fa72ed5a8da3bfc0f770461
Author: Tony Cook <tony at develop-help.com>
Date:   Wed Aug 12 16:20:16 2020 +1000

    Data::Dumper (XS): use mortals to prevent leaks if magic throws

    For example:

      use Tie::Scalar;
      use Data::Dumper;
      sub T::TIESCALAR { bless {}, shift}
      sub T::FETCH { die }
      my $x;
      tie $x, "T" or die;
      while(1) {
          eval { () = Dumper( [ \$x ] ) };
      }

    would leak various work SVs.

    I start a new scope (ENTER/LEAVE) for most recursive DD_dump() calls
    so that the work SVs don't accumulate on the temps stack, for example
    if we're dumping a large array we'd end up with several SVs on the
    temp stack for each member of the array.

    The exceptions are where I don't expect a large number of unreleased
    temps to accumulate, as with scalar or glob refs.

    (cherry picked from commit 815b4be4ab7ae210f796fc9d29754e55fc0d1f0e)

commit cd55a14f8579ba68f5f508018eafc559692b7a5e
Author: Tony Cook <tony at develop-help.com>
Date:   Tue Aug 11 10:46:38 2020 +1000

    make postav and valstr mortal so they're freed soonish

    these can leak if the value being dumped (or any part of it)
    had get magic and that magic throws an exception.

    Several other SVs can also leak in that case, but cleaning those up
    is more complex.

    (cherry picked from commit b98a3a6d08f681353d0b357fd1cce437c93656e7)

commit 1e63714900e7a528d0567ad241691c80439ffb9c
Author: Tony Cook <tony at develop-help.com>
Date:   Mon Aug 10 16:26:30 2020 +1000

    Data::Dumper: don't leak the working retval

    do this by mortalizing the SV on creation, rather than when we
    push it on the stack

    (cherry picked from commit 41463160be4baa0d81d9d8297508a1b9bdcaa206)

commit 5a4edcc643697c4815b845635cc938c367a056fa
Author: Karl Williamson <khw at cpan.org>
Date:   Wed Aug 12 17:53:52 2020 -0600

    Fix leak GH #18054

    This was a simple matter of one path failing to free the memory.

    (cherry picked from commit 6841cd5977c2d35ad75233734c66983a65613fce)

commit 776be7c3edd492f9b471f722b6f709e3af53c6bd
Author: Tony Cook <tony at develop-help.com>
Date:   Mon Apr 15 15:23:32 2019 +1000

    (perl #17844) don't update SvCUR until after we've done moving

    SvCUR() before the SvGROW() calls could result in reading beyond the
    end of a buffer.

    It wasn't a problem in the normal case, since sv_grow() just calls
    realloc() which has its own notion of how big the memory block is, but
    if the SV is SvOOK() sv_backoff() tries to move SvCUR()+1 bytes, which
    might be larger than the currently allocated size of the PV.

    (cherry picked from commit 45f235c116d4deab95c576aff77fe46d609f8553)