[Pkg-openssl-devel] Bug#588267: openssl randomly get into endless loop when using pipe output and limited read in sub process

[Gabriel CORRE]

Package: openssl
Version: 0.9.8g-15+lenny7
Severity: normal

This issue will not append not all the time.
I have got success and failure with the same keyfile, salt and input data.
When the issue appends, openssl seems enter in endless loop.

I use the -debug option to see :
1) without the issue

BEGIN
BIO[080BE738]:ctrl(106) - FILE pointer
BIO[080BE738]:ctrl return 1
BIO[080CC110]:ctrl(106) - FILE pointer
BIO[080CC110]:ctrl return 1
BIO[080CC110]:write(0,8) - FILE pointer
BIO[080CC110]:write return 8
BIO[080CC110]:write(0,8) - FILE pointer
BIO[080CC110]:write return 8
BIO[080CC2C0]:ctrl(6) - cipher
BIO[080CC110]:ctrl(6) - FILE pointer
BIO[080CC110]:ctrl return 0
BIO[080CC2C0]:ctrl return 0
BIO[080BE738]:read(0,32768) - FILE pointer
BIO[080BE738]:read return 512
BIO[080CC2C0]:write(0,512) - cipher
BIO[080CC110]:write(0,512) - FILE pointer
BIO[080CC110]:write return 512
BIO[080CC2C0]:write return 512
BIO[080BE738]:read(0,32768) - FILE pointer
BIO[080BE738]:read return 0
BIO[080CC2C0]:ctrl(11) - cipher
BIO[080CC110]:write(0,16) - FILE pointer
BIO[080CC110]:write return 16
BIO[080CC110]:ctrl(11) - FILE pointer
BIO[080CC110]:ctrl return 1
BIO[080CC2C0]:ctrl return 1
BIO[080BE738]:Free - FILE pointer
BIO[080CC110]:Free - FILE pointer
BIO[080CC2C0]:Free - cipher
END

2) with the issue

BEGIN
BIO[080BE738]:ctrl(106) - FILE pointer
BIO[080BE738]:ctrl return 1
BIO[080CC110]:ctrl(106) - FILE pointer
BIO[080CC110]:ctrl return 1
BIO[080CC110]:write(0,8) - FILE pointer
BIO[080CC110]:write return 8
BIO[080CC110]:write(0,8) - FILE pointer
BIO[080CC110]:write return 8
BIO[080CC2C0]:ctrl(6) - cipher
BIO[080CC110]:ctrl(6) - FILE pointer
BIO[080CC110]:ctrl return 0
BIO[080CC2C0]:ctrl return 0
BIO[080BE738]:read(0,32768) - FILE pointer
BIO[080BE738]:read return 512
BIO[080CC2C0]:write(0,512) - cipher
BIO[080CC110]:write(0,512) - FILE pointer
BIO[080CC110]:write return 0
BIO[080CC2C0]:write return 512
BIO[080BE738]:read(0,32768) - FILE pointer
BIO[080BE738]:read return 0
BIO[080CC2C0]:ctrl(11) - cipher
BIO[080CC110]:write(0,512) - FILE pointer
BIO[080CC110]:write return 0
BIO[080CC110]:write(0,512) - FILE pointer
BIO[080CC110]:write return 0
BIO[080CC110]:write(0,512) - FILE pointer
BIO[080CC110]:write return 0
BIO[080CC110]:write(0,512) - FILE pointer
BIO[080CC110]:write return 0
BIO[080CC110]:write(0,512) - FILE pointer
BIO[080CC110]:write return 0
BIO[080CC110]:write(0,512) - FILE pointer
BIO[080CC110]:write return 0
BIO[080CC110]:write(0,512) - FILE pointer
BIO[080CC110]:write return 0
BIO[080CC110]:write(0,512) - FILE pointer
BIO[080CC110]:write return 0
BIO[080CC110]:write(0,512) - FILE pointer
BIO[080CC110]:write return 0
BIO[080CC110]:write(0,512) - FILE pointer
BIO[080CC110]:write return 0
BIO[080CC110]:write(0,512) - FILE pointer
BIO[080CC110]:write return 0
BIO[080CC110]:write(0,512) - FILE pointer
BIO[080CC110]:write return 0
BIO[080CC110]:write(0,512) - FILE pointer
BIO[080CC110]:write return 0
BIO[080CC110]:write(0,512) - FILE pointer
BIO[080CC110]:write return 0
BIO[080CC110]:write(0,512) - FILE pointer
BIO[080CC110]:write return 0
BIO[080CC110]:write(0,512) - FILE pointer
[truncated]

My script to reproduce the issue :

#!/bin/sh

KEYFILE=/tmp/test.fakeprivatekey
TMPSALT=/tmp/test.salt
TMP=/tmp/test.head
RANDOMDATA=/tmp/test.random.data

if [ ! -f "$KEYFILE" ]; then
        dd if=/dev/urandom bs=1 count=512 of="$KEYFILE"
        echo "KEYFILE $KEYFILE created"
fi
if [ ! -f "$TMPSALT" ]; then
        echo -n '06c69ac405593b94' > "$TMPSALT"
        echo "TMPSALT $TMPSALT created"
fi
if [ ! -f "$RANDOMDATA" ]; then
        dd if=/dev/urandom obs=32768 count=1 2>/dev/null > "$RANDOMDATA"
        echo "RANDOMDATA $RANDOMDATA created"
fi

echo "BEGIN check"

# issue append a lot by this way :
openssl aes-256-cbc -debug -bufsize 32768 -pass file:$KEYFILE -salt -S "`cat $TMPSALT`" < $RANDOMDATA \
| hexdump -n 16 > "$TMP"

# issue not found by this way :
#openssl aes-256-cbc -debug -bufsize 32768 -pass file:$KEYFILE -salt -S "`cat $TMPSALT`" < $RANDOMDATA > $TMP2
#cat $TMP2 | hexdump -n 16 > "$TMP"

echo "END check"



-- System Information:
Debian Release: 5.0.4
  APT prefers stable
  APT policy: (500, 'stable')
Architecture: i386 (i686)

Kernel: Linux 2.6.26-2-686 (SMP w/2 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash

Versions of packages openssl depends on:
ii  libc6                  2.7-18lenny4      GNU C Library: Shared libraries
ii  libssl0.9.8            0.9.8g-15+lenny6  SSL shared libraries
ii  zlib1g                 1:1.2.3.3.dfsg-12 compression library - runtime

openssl recommends no packages.

Versions of packages openssl suggests:
ii  ca-certificates               20080809   Common CA certificates

-- no debconf information