Bug#863636: diffoscope: usage of FIFOs causes pair-comparisons to not run in parallel, wasting performance by about 1/2

[Ximin Luo]

Package: diffoscope
Version: 78
Severity: normal

Dear Maintainer,

diff(1) first reads the contents of one file then the next one:

https://sources.debian.net/src/diffutils/1:3.5-3/src/io.c/#L552

This means that if the "files" are actually FIFOs connected to the output of a
process, as they are in many cases in diffoscope, the second process has to wait
for diff(1) to fully read the output of the first process, before it itself can
run. This prevents both processes from running in parallel.

An appropriate fix would be to store the output of at least one of the commands
into a temporary file, and have diff(1) read from this instead. This has to be
done carefully however, to make sure that diff(1) doesn't accidentally read it
before the process is finished.

X

-- System Information:
Debian Release: 9.0
  APT prefers testing
  APT policy: (990, 'testing'), (500, 'unstable-debug'), (500, 'testing-debug'), (500, 'buildd-unstable'), (300, 'unstable'), (100, 'experimental'), (1, 'experimental-debug')
Architecture: amd64
 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.9.0-3-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_GB.utf8, LC_CTYPE=en_GB.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages diffoscope depends on:
ii  python3-libarchive-c   2.1-3.1
ii  python3-magic          1:5.30-1
ii  python3-pkg-resources  33.1.1-1
pn  python3:any            <none>

Versions of packages diffoscope recommends:
ii  acl                              2.2.52-3+b1
ii  apktool                          2.2.1+dfsg-2
ii  binutils-multiarch               2.28-5
ii  bzip2                            1.0.6-8.1
ii  caca-utils                       0.99.beta19-2+b2
ii  colord                           1.3.3-2
ii  default-jdk [java-sdk]           2:1.8-58
ii  default-jdk-headless             2:1.8-58
ii  enjarify                         1:1.0.3-3
ii  fontforge-extras                 0.3-4
pn  fp-utils                         <none>
ii  genisoimage                      9:1.1.11-3+b2
ii  gettext                          0.19.8.1-2
ii  ghc                              8.0.1-17+b1
ii  ghostscript                      9.20~dfsg-3.2
ii  gnupg                            2.1.18-6
ii  imagemagick                      8:6.9.7.4+dfsg-8
ii  imagemagick-6.q16 [imagemagick]  8:6.9.7.4+dfsg-8
ii  jsbeautifier                     1.6.4-6
ii  llvm                             1:3.8-36
ii  mono-utils                       4.6.2.7+dfsg-1
ii  openjdk-8-jdk [java-sdk]         8u131-b11-2
ii  openssh-client                   1:7.4p1-10
ii  pdftk                            2.02-4+b2
ii  poppler-utils                    0.48.0-2
ii  python3-argcomplete              1.8.1-1
ii  python3-debian                   0.1.30
pn  python3-guestfs                  <none>
ii  python3-progressbar              2.3-4
ii  python3-rpm                      4.12.0.2+dfsg1-2
ii  python3-tlsh                     3.4.4+20151206-1+b2
ii  rpm2cpio                         4.12.0.2+dfsg1-2
ii  sng                              1.1.0-1+b1
ii  sqlite3                          3.16.2-3
ii  squashfs-tools                   1:4.3-3+b1
ii  unzip                            6.0-21
ii  vim-common                       2:8.0.0197-4
ii  xxd                              2:8.0.0197-4
ii  xz-utils                         5.2.2-1.2+b1

Versions of packages diffoscope suggests:
ii  libjs-jquery  3.1.1-2

-- no debconf information