Bug#611195: vlc-nox: generating plugin cache at postinst segfaults on PaX kernel

[pageexec at freemail.hu]

On 26 Jan 2011 at 17:03, Pierre Ynard wrote:

> mprotect(0x4946b000, 3840, PROT_READ|PROT_WRITE) = -1 EACCES (Permission denied)
> --- SIGSEGV (Segmentation fault) @ 0 (0) ---
> +++ killed by SIGSEGV +++
> zsh: segmentation fault  strace -f /usr/lib/vlc/vlc-cache-gen /usr/lib/vlc/plugins
> 
> Note the mprotect() error, as logged by the kernel:
> 
> [1607219.198034] grsec: denied RWX mprotect of /lib/ld-2.11.2.so by
> /usr/lib/vlc/vlc-cache-gen[vlc-cache-gen:15667] uid/euid:0/0 gid/egid:0/0,
> parent /var/lib/dpkg/info/vlc-nox.postinst[vlc-nox.postins:15666] uid/euid:0/0 gid/egid:0/0
> 
> Disabling the mprotect() restriction feature makes the problem go away.
> 
> [...]
>
> Feel free to forward the bug appropriately, but:
> 
>  - Why is the dynamic loader crashing, instead of handling the error
>    gracefully?
>  - What's up with libSDL-1.2?
>  - It sucks that installation and/or start-up of VLC fails because of
>    one faulty module

this is a long standing bug in glibc and RELRO handling in ld.so. what happens is this:

1. ld.so maintains the current stack execution rights in a variable (most of the time
   they'd be rw-, i.e., non-executable)
2. said variable is marked to be stored in the RELRO segment of ld.so
3. when a dynamically linked program is started, ld.so performs all the relocations, etc
   then as a final step it mprotects the RELRO segments of all binaries (including itself)
   as actually read-only
4. PaX/MPROTECT enforces this read-only property in that it denies further mprotect
   requests that would make any part of such RELRO segments writable again (which is
   kinda the whole idea behind RELRO)
5. later a library is dlopen'd that wants an executable stack (has a RWE GNU_STACK segment)
   so ld.so wants to update this internal variable as well but since it's in the RELRO
   segment, it makes that area writable temporarily. this step fails per 4 and since the
   mprotect return value is not checked, the attempt of actually writing to the variable
   will cause a segfault.

now as you can see there's a cascade of failures here. the fundamental issue is the
whole broken concept of GNU_STACK but i guess debian folks are not about to neutralize
it, so let's look at the next step: putting a writable variable into the RELRO segment.

it was originally done in the hope that an exploit wouldn't be able to modify it either
and therefore prevent a specific ret2libc class of attack that'd abuse ld.so to do the
attacker's bidding by creating an rwx stack (let's not digress into how pointless this
is). this would work if this variable didn't need to be modified *after* RELRO enforcement
had taken place, but RWE GNU_STACK support for dlopen'd libraries requires exactly that
(which of course also circumvents GNU_STACK but let's not digress again ;). so what glibc
devs decided to do was to temporarily revert the RELRO segment's read-only status to update
the variable but for some reason they didn't check the return value of mprotect (probably
on grounds that 'it cannot fail' since it's called on a known valid region, etc except
there're many other reasons that can make mprotect fail).

so there you have it in a nutshell. probably the quickest 'fix' is to at least add a check
for mprotect's return value but that still won't let this libSDL library to load under PaX
and MPROTECT (whether it really needs an executable stack or is just incorrectly marked due
to some unmarked .S files is another angle you can investigate).

cheers,

 PaX Team