Bug#1107564: unblock: diffoscope/297
Chris Lamb
lamby at debian.org
Mon Jun 9 18:39:01 BST 2025
Package: release.debian.org
User: release.debian.org at packages.debian.org
Usertags: unblock
X-Debbugs-CC: reproducible-builds at lists.alioth.debian.org
Dear Release Team,
Please consider unblocking diffoscope 297. This is principally to fix
the use of potentially dangerous functionality in zipdetails
(addressed in diffoscope versions 295 and 206).
Regrettably, a new upload (297) was then made that introduced lzma
comparator support, and NuGet package support was included in 296. We
did not have our mind on the freeze, alas...
However, I foresee no regressions here, as new functionality like this
is fairly well compartmentalised in diffoscope, thus I think it is
safe to include — and not require first rolling back these changes or
similar.
The full debdiff is attached, but the relevant changelog entries are
included inline here:
diffoscope (297) unstable; urgency=medium
[ Will Hollywood ]
* Add a LZMA comparator and tests.
-- Chris Lamb <lamby at debian.org> Fri, 30 May 2025 09:34:34 -0700
diffoscope (296) unstable; urgency=medium
[ Chris Lamb ]
* Don't rely on zipdetails' --walk functionality to be available; only add
that argument after testing for a new enough versions.
(Closes: reproducible-builds/diffoscope#408)
* Disable and then re-enable failing on stable-bpo.
* Update copyright years.
[ Omair Majid ]
* Add NuGet package support.
-- Chris Lamb <lamby at debian.org> Fri, 16 May 2025 08:41:37 -0700
diffoscope (295) unstable; urgency=medium
[ Chris Lamb ]
* Use --walk over the potentially dangerous --scan argument of zipdetails(1).
(Closes: reproducible-builds/diffoscope#406)
-- Chris Lamb <lamby at debian.org> Fri, 09 May 2025 09:13:39 -0700
Best wishes,
--
,''`.
: :' : Chris Lamb
`. `'` lamby at debian.org / chris-lamb.co.uk
`-
-------------- next part --------------
diff --git debian/changelog debian/changelog
index 47a350bf..95507593 100644
--- debian/changelog
+++ debian/changelog
@@ -1,3 +1,32 @@
+diffoscope (297) unstable; urgency=medium
+
+ [ Will Hollywood ]
+ * Add a LZMA comparator and tests.
+
+ -- Chris Lamb <lamby at debian.org> Fri, 30 May 2025 09:34:34 -0700
+
+diffoscope (296) unstable; urgency=medium
+
+ [ Chris Lamb ]
+ * Don't rely on zipdetails' --walk functionality to be available; only add
+ that argument after testing for a new enough versions.
+ (Closes: reproducible-builds/diffoscope#408)
+ * Disable and then re-enable failing on stable-bpo.
+ * Update copyright years.
+
+ [ Omair Majid ]
+ * Add NuGet package support.
+
+ -- Chris Lamb <lamby at debian.org> Fri, 16 May 2025 08:41:37 -0700
+
+diffoscope (295) unstable; urgency=medium
+
+ [ Chris Lamb ]
+ * Use --walk over the potentially dangerous --scan argument of zipdetails(1).
+ (Closes: reproducible-builds/diffoscope#406)
+
+ -- Chris Lamb <lamby at debian.org> Fri, 09 May 2025 09:13:39 -0700
+
diffoscope (294) unstable; urgency=medium
[ Chris Lamb ]
diff --git diffoscope/__init__.py diffoscope/__init__.py
index 93b0f258..f7a303d4 100644
--- diffoscope/__init__.py
+++ diffoscope/__init__.py
@@ -17,4 +17,4 @@
# You should have received a copy of the GNU General Public License
# along with diffoscope. If not, see <https://www.gnu.org/licenses/>.
-VERSION = "294"
+VERSION = "297"
diff --git diffoscope/comparators/__init__.py diffoscope/comparators/__init__.py
index 6621079d..1c51d080 100644
--- diffoscope/comparators/__init__.py
+++ diffoscope/comparators/__init__.py
@@ -82,6 +82,7 @@ class ComparatorManager:
("java.ClassFile",),
("lz4.Lz4File",),
("lzip.LzipFile",),
+ ("lzma.LzmaFile",),
("mono.MonoExeFile",),
("pdf.PdfFile",),
("png.PngFile",),
@@ -98,6 +99,7 @@ class ComparatorManager:
("ocaml.OcamlInterfaceFile",),
("docx.DocxFile",),
("zip.MozillaZipFile",),
+ ("zip.NuGetPackageFile",),
("zip.JmodJavaModule",),
("zip.ZipFile",),
("image.JPEGImageFile",),
diff --git diffoscope/comparators/lzma.py diffoscope/comparators/lzma.py
new file mode 100644
index 00000000..8836f98a
--- /dev/null
+++ diffoscope/comparators/lzma.py
@@ -0,0 +1,65 @@
+#
+# diffoscope: in-depth comparison of files, archives, and directories
+#
+# Copyright © 2014-2015 Jérémy Bobbio <lunar at debian.org>
+# Copyright © 2015-2020, 2024-2025 Chris Lamb <lamby at debian.org>
+#
+# diffoscope is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# diffoscope is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with diffoscope. If not, see <https://www.gnu.org/licenses/>.
+
+import re
+import logging
+import subprocess
+
+from diffoscope.tools import tool_required
+
+
+from .utils.file import File
+from .utils.archive import Archive
+
+logger = logging.getLogger(__name__)
+
+
+class LzmaContainer(Archive):
+ def open_archive(self):
+ return self
+
+ def close_archive(self):
+ pass
+
+ def get_member_names(self):
+ return [self.get_compressed_content_name(".lzma")]
+
+ @tool_required("xz")
+ def extract(self, member_name, dest_dir):
+ dest_path = self.get_path_name(dest_dir)
+ logger.debug("lzma extracting to %s", dest_path)
+ with open(dest_path, "wb") as fp:
+ subprocess.check_call(
+ [
+ "xz",
+ "--decompress",
+ "--format=lzma",
+ "--stdout",
+ self.source.path,
+ ],
+ stdout=fp,
+ stderr=None,
+ )
+ return dest_path
+
+
+class LzmaFile(File):
+ DESCRIPTION = "LZMA compressed files"
+ CONTAINER_CLASSES = [LzmaContainer]
+ FILE_TYPE_RE = re.compile(r"^LZMA compressed data\b")
diff --git diffoscope/comparators/zip.py diffoscope/comparators/zip.py
index 5801e37d..80dc85ae 100644
--- diffoscope/comparators/zip.py
+++ diffoscope/comparators/zip.py
@@ -32,7 +32,11 @@ from diffoscope.tempfiles import get_named_temporary_file
from .utils.file import File
from .directory import Directory
from .utils.archive import Archive, ArchiveMember
-from .utils.command import Command
+from .utils.command import Command, our_check_output
+
+
+def zipdetails_version():
+ return our_check_output(["zipdetails", "--version"]).decode("UTF-8")
class Zipinfo(Command):
@@ -157,7 +161,19 @@ def zipinfo_differences(file, other):
class Zipdetails(Command):
@tool_required("zipdetails")
def cmdline(self):
- return ["zipdetails", "--redact", "--scan", "--utc", self.path]
+ # See <https://salsa.debian.org/reproducible-builds/diffoscope/-/issues/406>
+ # for discussion of zipdetails command line arguments.
+ #
+ # Older zipdetails does not support --walk; added in Debian
+ # 5.40.0~rc1-1, but "zipdetails --version" shipped in, say, perl
+ # 5.36.0-7+deb12u1 returns "2.104".
+ #
+ # See <https://salsa.debian.org/reproducible-builds/diffoscope/-/issues/408>
+ #
+ if float(zipdetails_version()) < 4.0:
+ return ["zipdetails", "--redact", "--utc", self.path]
+
+ return ["zipdetails", "--redact", "--walk", "--utc", self.path]
class ZipDirectory(Directory, ArchiveMember):
@@ -340,6 +356,13 @@ class MozillaZipFile(ZipFile):
return file.file_header[4:8] == b"PK\x01\x02"
+class NuGetPackageFile(ZipFile):
+ DESCRIPTION = "NuGet packages"
+ CONTAINER_CLASSES = [ZipContainer]
+ FILE_TYPE_HEADER_PREFIX = b"PK\x03\x04"
+ FILE_EXTENSION_SUFFIX = {".nupkg"}
+
+
class JmodJavaModule(ZipFile):
DESCRIPTION = "Java .jmod modules"
FILE_TYPE_RE = re.compile(r"^(Zip archive data|Java jmod module)")
diff --git diffoscope/external_tools.py diffoscope/external_tools.py
index 3d698153..5abcae99 100644
--- diffoscope/external_tools.py
+++ diffoscope/external_tools.py
@@ -132,6 +132,7 @@ EXTERNAL_TOOLS = {
},
"lz4": {"debian": "lz4", "FreeBSD": "lz4", "guix": "lz4"},
"lzip": {"debian": "lzip", "guix": "lzip"},
+ "lzma": {"debian": "xz-utils", "arch": "xz", "guix": "xz"},
"msgunfmt": {
"debian": "gettext",
"arch": "gettext",
diff --git tests/comparators/test_lzma.py tests/comparators/test_lzma.py
new file mode 100644
index 00000000..696da3f0
--- /dev/null
+++ tests/comparators/test_lzma.py
@@ -0,0 +1,75 @@
+#
+# diffoscope: in-depth comparison of files, archives, and directories
+#
+# Copyright © 2015 Jérémy Bobbio <lunar at debian.org>
+# Copyright © 2016-2017, 2020, 2024-2025 Chris Lamb <lamby at debian.org>
+#
+# diffoscope is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# diffoscope is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with diffoscope. If not, see <https://www.gnu.org/licenses/>.
+
+import shutil
+import pytest
+
+from diffoscope.comparators.lzma import LzmaFile
+from diffoscope.comparators.binary import FilesystemFile
+from diffoscope.comparators.utils.specialize import specialize
+
+from ..utils.data import load_fixture, assert_diff
+from ..utils.tools import skip_unless_tools_exist
+from ..utils.nonexisting import assert_non_existing
+
+lzma1 = load_fixture("test1.lzma")
+lzma2 = load_fixture("test2.lzma")
+
+
+def test_identification(lzma1):
+ assert isinstance(lzma1, LzmaFile)
+
+
+def test_no_differences(lzma1):
+ difference = lzma1.compare(lzma1)
+ assert difference is None
+
+
+ at pytest.fixture
+def differences(lzma1, lzma2):
+ return lzma1.compare(lzma2).details
+
+
+ at skip_unless_tools_exist("xz")
+def test_content_source(differences):
+ assert differences[0].source1 == "test1"
+ assert differences[0].source2 == "test2"
+
+
+ at skip_unless_tools_exist("xz")
+def test_content_source_without_extension(tmpdir, lzma1, lzma2):
+ path1 = str(tmpdir.join("test1"))
+ path2 = str(tmpdir.join("test2"))
+ shutil.copy(lzma1.path, path1)
+ shutil.copy(lzma2.path, path2)
+ lzma1 = specialize(FilesystemFile(path1))
+ lzma2 = specialize(FilesystemFile(path2))
+ difference = lzma1.compare(lzma2).details
+ assert difference[0].source1 == "test1-content"
+ assert difference[0].source2 == "test2-content"
+
+
+ at skip_unless_tools_exist("xz")
+def test_content_diff(differences):
+ assert_diff(differences[0], "text_ascii_expected_diff")
+
+
+ at skip_unless_tools_exist("xz")
+def test_compare_non_existing(monkeypatch, lzma1):
+ assert_non_existing(monkeypatch, lzma1)
diff --git tests/comparators/test_zip.py tests/comparators/test_zip.py
index 303b6f80..a5538c7b 100644
--- tests/comparators/test_zip.py
+++ tests/comparators/test_zip.py
@@ -2,7 +2,7 @@
# diffoscope: in-depth comparison of files, archives, and directories
#
# Copyright © 2015 Jérémy Bobbio <lunar at debian.org>
-# Copyright © 2015-2020, 2022, 2024 Chris Lamb <lamby at debian.org>
+# Copyright © 2015-2020, 2022, 2024-2025 Chris Lamb <lamby at debian.org>
#
# diffoscope is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
@@ -20,7 +20,13 @@
import pytest
import subprocess
-from diffoscope.comparators.zip import ZipFile, MozillaZipFile, JmodJavaModule
+from diffoscope.comparators.zip import (
+ ZipFile,
+ MozillaZipFile,
+ NuGetPackageFile,
+ JmodJavaModule,
+ zipdetails_version,
+)
from ..utils.data import load_fixture, assert_diff
from ..utils.tools import skip_unless_tools_exist, skip_unless_tool_is_at_least
@@ -34,16 +40,14 @@ encrypted_zip1 = load_fixture("encrypted1.zip")
encrypted_zip2 = load_fixture("encrypted2.zip")
mozzip1 = load_fixture("test1.mozzip")
mozzip2 = load_fixture("test2.mozzip")
+nupkg1 = load_fixture("test1.nupkg")
+nupkg2 = load_fixture("test2.nupkg")
jmod1 = load_fixture("test1.jmod")
jmod2 = load_fixture("test2.jmod")
test_comment1 = load_fixture("test_comment1.zip")
test_comment2 = load_fixture("test_comment2.zip")
-def zipdetails_version():
- return subprocess.check_output(["zipdetails", "--version"]).decode("UTF-8")
-
-
def io_compress_zip_version():
try:
return subprocess.check_output(
@@ -135,6 +139,42 @@ def test_mozzip_compare_non_existing(monkeypatch, mozzip1):
assert_non_existing(monkeypatch, mozzip1)
+def test_nupkg_identification(nupkg1):
+ assert isinstance(nupkg1, NuGetPackageFile)
+
+
+def test_nupkg_no_differences(nupkg1):
+ difference = nupkg1.compare(nupkg1)
+ assert difference is None
+
+
+ at pytest.fixture
+def nupkg_differences(nupkg1, nupkg2):
+ return nupkg1.compare(nupkg2).details
+
+
+ at skip_unless_tools_exist("zipinfo")
+def test_nupkg_metadata(nupkg_differences, nupkg1, nupkg2):
+ assert_diff(nupkg_differences[0], "nupkg_expected_diff")
+
+
+ at skip_unless_tools_exist("zipinfo")
+def test_nupkg_compressed_files(nupkg_differences):
+ assert (
+ nupkg_differences[-1].source1
+ == "package/services/metadata/core-properties/b44ebb537bbf4983b9527f9e3820fda6.psmdcp"
+ )
+ assert (
+ nupkg_differences[-1].source2
+ == "package/services/metadata/core-properties/08f1f9d8789a4668a128f78560bd0107.psmdcp"
+ )
+
+
+ at skip_unless_tools_exist("zipinfo")
+def test_nupkg_compare_non_existing(monkeypatch, nupkg1):
+ assert_non_existing(monkeypatch, nupkg1)
+
+
def test_jmod_identification(jmod1):
assert isinstance(jmod1, JmodJavaModule)
diff --git tests/data/nupkg_expected_diff tests/data/nupkg_expected_diff
new file mode 100644
index 00000000..c8b39ff9
--- /dev/null
+++ tests/data/nupkg_expected_diff
@@ -0,0 +1,11 @@
+@@ -1,7 +1,7 @@
+-Zip file size: 3163 bytes, number of entries: 5
++Zip file size: 3162 bytes, number of entries: 5
+ -rw-r--r-- 2.0 unx 502 b- defN 25-Apr-14 18:14 _rels/.rels
+ -rw-r--r-- 2.0 unx 407 b- defN 25-Apr-14 18:14 ClassLibrary.nuspec
+ -rw-r--r-- 2.0 unx 3584 b- defN 25-Apr-14 22:14 lib/net8.0/ClassLibrary.dll
+ -rw-r--r-- 2.0 unx 459 b- defN 25-Apr-14 18:14 [Content_Types].xml
+--rw-r--r-- 2.0 unx 625 b- defN 25-Apr-14 18:14 package/services/metadata/core-properties/b44ebb537bbf4983b9527f9e3820fda6.psmdcp
+-5 files, 5577 bytes uncompressed, 2447 bytes compressed: 56.1%
++-rw-r--r-- 2.0 unx 625 b- defN 25-Apr-14 18:14 package/services/metadata/core-properties/08f1f9d8789a4668a128f78560bd0107.psmdcp
++5 files, 5577 bytes uncompressed, 2446 bytes compressed: 56.1%
diff --git tests/data/test1.lzma tests/data/test1.lzma
new file mode 100644
index 00000000..d7186e5d
Binary files /dev/null and tests/data/test1.lzma differ
diff --git tests/data/test1.nupkg tests/data/test1.nupkg
new file mode 100644
index 00000000..c1b85928
Binary files /dev/null and tests/data/test1.nupkg differ
diff --git tests/data/test2.lzma tests/data/test2.lzma
new file mode 100644
index 00000000..28d7d0d6
Binary files /dev/null and tests/data/test2.lzma differ
diff --git tests/data/test2.nupkg tests/data/test2.nupkg
new file mode 100644
index 00000000..c439e5b8
Binary files /dev/null and tests/data/test2.nupkg differ
More information about the Reproducible-builds
mailing list