Bug#745823: libwww-perl: an https request with iso-8859-1 headers, chunked transfer and data with utf8 bit on is corrupted.

[John Hughes]

On 25/04/14 22:01, Niko Tyni wrote:
> found 745823 6.06-1
> thanks

My pleasure.
> Interesting. I can reproduce this on (mostly current) sid with
> libwww-perl 6.06-1.

Ah, I was going to test that Monday :-)
> Quoting HTTP::Request documentation:
>
>       $r->content( $bytes )
>
>             Note that the content should be a string of bytes.  Strings in
>             perl can contain characters outside the range of a byte.
>             The "Encode" module can be used to turn such strings into a
>             string of bytes.
>
> So this is not totally unexpected, but the particular failure mode you've
> run into is certainly rather horrible.
>
> Possibly the content() method should croak when the UTF8 bit is set?
Interestingly in the my case, although the UTF8 bit is set, the data is 
all code points below 256.  In fact the first time i ran into the bug 
the data was "XXX".  (Read from a file with "binmode :utf8" on).

Maybe something like

     if (utf8::is_utf8($data)) {
         eval {
             utf8::downgrade ($data);
         };
         croak "content not bytes" if $@;
     }