[med-svn] [Git][med-team/conda-package-handling][master] 5 commits: Fix watch URL

Nilesh Patra gitlab at salsa.debian.org
Tue Apr 20 12:43:20 BST 2021



Nilesh Patra pushed to branch master at Debian Med / conda-package-handling


Commits:
3061d420 by Nilesh Patra at 2021-04-20T17:06:05+05:30
Fix watch URL

- - - - -
7ca4d69c by Nilesh Patra at 2021-04-20T17:06:22+05:30
New upstream version 1.7.3
- - - - -
baddcfa6 by Nilesh Patra at 2021-04-20T17:06:23+05:30
Update upstream source from tag 'upstream/1.7.3'

Update to upstream version '1.7.3'
with Debian dir ae721309b09b1768daae568bc7c334915dc2dba5
- - - - -
05c3ec09 by Nilesh Patra at 2021-04-20T17:09:11+05:30
Remove merged patch

- - - - -
844e46f1 by Nilesh Patra at 2021-04-20T17:11:52+05:30
Interim changelog entry

- - - - -


19 changed files:

- .authors.yml
- + .github/workflows/CI.yaml
- .mailmap
- .travis.yml
- AUTHORS.txt
- CHANGELOG.md
- appveyor.yml
- conda.recipe/meta.yaml
- debian/changelog
- − debian/patches/fix-out-dir.patch
- debian/patches/series
- debian/watch
- + devtools/conda-envs/test_env.yaml
- src/conda_package_handling/_version.py
- src/conda_package_handling/api.py
- src/conda_package_handling/archive_utils_c.c
- src/conda_package_handling/cli.py
- src/conda_package_handling/tarball.py
- tests/test_api.py


Changes:

=====================================
.authors.yml
=====================================
@@ -25,10 +25,26 @@
   github: alanhdu
 - name: Ray Donnelly
   email: mingw.android at gmail.com
-  num_commits: 7
+  num_commits: 14
   first_commit: 2020-05-06 10:49:55
+  github: mingwandroid
 - name: ossdev07
   email: ossdev at puresoftware.com
   num_commits: 1
   first_commit: 2020-01-06 10:18:31
   github: ossdev07
+- name: leej3
+  email: johnleenimh at gmail.com
+  num_commits: 12
+  first_commit: 2021-02-23 12:20:16
+  github: leej3
+- name: Matthew R. Becker
+  email: beckermr at users.noreply.github.com
+  num_commits: 2
+  first_commit: 2020-12-29 14:58:11
+  github: beckermr
+- name: Eli Uriegas
+  email: seemethere101 at gmail.com
+  num_commits: 1
+  first_commit: 2021-03-14 08:41:18
+  github: seemethere


=====================================
.github/workflows/CI.yaml
=====================================
@@ -0,0 +1,63 @@
+name: CI
+
+on:
+  # GitHub has started calling new repo's first branch "main" https://github.com/github/renaming
+  # Existing codes likely still have "master" as the primary branch
+  # Both are tracked here to keep legacy and new codes working
+  push:
+    branches:
+      - "master"
+      - "main"
+  pull_request:
+    branches:
+      - "master"
+      - "main"
+
+jobs:
+  test:
+    name: Test on ${{ matrix.os }}, Python ${{ matrix.python-version }}
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os: [ubuntu-latest, windows-latest]
+        python-version: [3.7, 3.8, 3.9]
+
+    steps:
+    - uses: actions/checkout at v1
+
+    - name: Additional info about the build
+      shell: bash
+      run: |
+        uname -a
+        df -h
+        ulimit -a
+
+
+    # More info on options: https://github.com/conda-incubator/setup-miniconda
+    - uses: conda-incubator/setup-miniconda at v2
+      with:
+        python-version: ${{ matrix.python-version }}
+        environment-file: devtools/conda-envs/test_env.yaml
+
+        channels: defaults
+
+        activate-environment: test_env
+        auto-update-conda: false
+        auto-activate-base: false
+        show-channel-urls: true
+
+    - name: Install package
+
+      # conda setup requires this special shell
+      shell: bash -l {0}
+      run: |
+        conda build conda.recipe
+
+# TODO: Re-enable codecov when we figure out how to grab coverage from the conda build
+#       environment
+#     - name: CodeCov
+#       uses: codecov/codecov-action at v1
+#       with:
+#         file: ./coverage.xml
+#         flags: unittests
+#         name: codecov-${{ matrix.os }}-py${{ matrix.python-version }}


=====================================
.mailmap
=====================================
@@ -11,8 +11,11 @@
 
 Michael Sarahan <msarahan at gmail.com> Mike Sarahan <msarahan at continuum.io>
 Michael Sarahan <msarahan at gmail.com> Mike Sarahan <msarahan at anaconda.com>
-Jonathan J. Helmus <jjhelmus at gmail.com>
 Ray Donnelly <mingw.android at gmail.com>
+leej3 <johnleenimh at gmail.com>
+Jonathan J. Helmus <jjhelmus at gmail.com>
 Nehal J Wani <nehaljw.kkd1 at gmail.com>
 Alan Du <alanhdu at gmail.com>
+Matthew R. Becker <beckermr at users.noreply.github.com>
 ossdev07 <ossdev at puresoftware.com>
+Eli Uriegas <seemethere101 at gmail.com>


=====================================
.travis.yml
=====================================
@@ -12,35 +12,42 @@ matrix:
 env:
   global:
     - SUDO=""
+    - CONDA_ALWAYS_YES=1
 install:
+  - set -e
   - if [[ "${TRAVIS_CPU_ARCH}" == "arm64" ]]; then
-      SUDO=sudo;
+      set -x;
+      # Archiconda install requires sudo for running commands. Pass the path
+      # through to the sudo environment so that condda is accessible
+      SUDO='sudo env PATH=$PATH'
       wget -q "https://github.com/Archiconda/build-tools/releases/download/0.2.3/Archiconda3-0.2.3-Linux-aarch64.sh" -O archiconda.sh;
       chmod +x archiconda.sh;
-      export CONDA_ALWAYS_YES=1;
       bash archiconda.sh -b -p $HOME/miniconda;
       export PATH="$HOME/miniconda/bin:$PATH";
-      $SUDO cp -r $HOME/miniconda/bin/* /usr/bin/;
-      $SUDO conda install python=3.7 conda conda-build pytest six pytest-cov pytest-mock;
+      $SUDO conda install python=3.7 conda conda-build;
+      set +x;
     else
-      SUDO="";
       wget https://repo.anaconda.com/pkgs/misc/conda-execs/conda-4.7.5-linux-64.exe -O conda.exe;
       chmod +x conda.exe;
-      export CONDA_ALWAYS_YES=1;
-      ./conda.exe create -p $HOME/miniconda python=$TRAVIS_PYTHON_VERSION conda conda-build pytest six pytest-cov pytest-mock;
-    fi
+      ./conda.exe create -p $HOME/miniconda python=$TRAVIS_PYTHON_VERSION conda conda-build;
+    fi;
   - export PATH="$HOME/miniconda/bin:$PATH"
   - hash -r
   - if [[ "$TRAVIS_PYTHON_VERSION" == "2.7" ]]; then
-      conda install futures;
-    fi
+      pip install futures mock pytest-cov pytest-mock;
+    else
+      pip3 install mock pytest-cov pytest-mock;
+    fi;
   - $SUDO conda build conda.recipe --no-test
   - $SUDO conda install --use-local conda-package-handling
   - $SUDO conda info -a
 script:
   # rebuilding the recipe with our new CPH installed tests it a bit deeper than the test suite.
+  - which conda
   - $SUDO conda build conda.recipe
-  - pytest -v --color=yes --cov=conda_package_handling tests
+  # the system pytest is used for coverage testing, cph from current conda envorinment is used.
+  - which pytest
+  - pytest -v --color=yes -rA --cov=conda_package_handling tests
 after_success:
-  - conda install codecov
+  - $SUDO conda install codecov
   - codecov


=====================================
AUTHORS.txt
=====================================
@@ -2,8 +2,11 @@ All of the people who have made at least one contribution to conda-package-handl
 Authors are sorted by number of commits.
 
 * Michael Sarahan
-* Jonathan J. Helmus
 * Ray Donnelly
+* leej3
+* Jonathan J. Helmus
 * Nehal J Wani
 * Alan Du
+* Matthew R. Becker
 * ossdev07
+* Eli Uriegas


=====================================
CHANGELOG.md
=====================================
@@ -1,4 +1,31 @@
 # current developments
+2021-04-12 1.7.3:
+------------------
+
+Enhancements:
+-------------
+
+* Python tar extraction now used as a fallback if libarchive fails
+
+Bug fixes:
+----------
+
+* Fix binsort's mangling of symlinks
+* Fix #71, larger directories fail to extract using libarchive
+* When testing that exceptions are raised or archives containing abs paths, first check that such a "broken" archive was created during test setup... otherwise skip the test.
+* api.create now raises an error correctly if archive creation failed or extension is not supported.
+* Travis CI issue now resolved, mock added as dependency for conda test environments and system dependencies
+* Fixed bug where extract parser cli failed due to not having ``out_folder`` attribute.
+
+Contributors:
+-------------
+
+* @mingwandroid
+* @leej3
+* @beckermr
+* @seemethere
+
+
 
 2020-10-16 1.7.2:
 ------------------


=====================================
appveyor.yml
=====================================
@@ -1,6 +1,6 @@
 environment:
   matrix:
-    - PYTHON_VERSION: "3.6"
+    - PYTHON_VERSION: "3.9"
     - PYTHON_VERSION: "2.7"
 
 install:


=====================================
conda.recipe/meta.yaml
=====================================
@@ -33,11 +33,12 @@ test:
     - pytest
     - pytest-cov
     - pytest-mock
+    - mock
   imports:
     - conda_package_handling
     - conda_package_handling.archive_utils_cy
   commands:
-    - pytest tests
+    - pytest -v --cov=tests/ --cov-report=xml --color=yes tests/
 about:
   home: https://github.com/conda/conda-package-handling
   summary: Create and extract conda packages of various formats


=====================================
debian/changelog
=====================================
@@ -1,3 +1,12 @@
+conda-package-handling (1.7.3-1) UNRELEASED; urgency=medium
+
+  * Team Upload.
+  * Fix watch URL
+  * New upstream version 1.7.3
+  * Remove merged patch
+
+ -- Nilesh Patra <nilesh at debian.org>  Tue, 20 Apr 2021 17:09:22 +0530
+
 conda-package-handling (1.7.2-2) unstable; urgency=medium
 
   * Team upload.


=====================================
debian/patches/fix-out-dir.patch deleted
=====================================
@@ -1,34 +0,0 @@
-Description: This seems like a regression in the new release hence adapted this function to the previous release.
- More details here: https://github.com/conda/conda-package-handling/issues/72
- Remove this patch if fixed in the next release.
-Author: Nilesh Patra <npatra974 at gmail.com>
-Forwarded: no
-Last-Update: 2020-10-28
---- a/src/conda_package_handling/cli.py
-+++ b/src/conda_package_handling/cli.py
-@@ -73,8 +73,6 @@
- 
- def main(args=None):
-     args = parse_args(args)
--    if args.out_folder:
--        args.out_folder = os.path.abspath(os.path.normpath(os.path.expanduser(args.out_folder))) + os.sep
-     if args.subparser_name in ('extract', 'x'):
-         if args.info:
-             api.extract(args.archive_path, args.dest, components='info', prefix=args.prefix)
-@@ -83,7 +81,7 @@
-     elif args.subparser_name in ('create', 'c'):
-         api.create(args.prefix, args.file_list, args.out_fn, args.out_folder)
-     elif args.subparser_name in ('transmute', 't'):
--        failed_files = api.transmute(args.in_file, args.out_ext, args.out_folder, args.processes or 1, force=args.force)
-+        failed_files = api.transmute(args.in_file, args.out_ext, args.out_folder, args.processes)
-         if failed_files:
-             print("failed files:")
-             pprint(failed_files)
-@@ -96,6 +94,7 @@
-             sys.exit(1)
-     else:
-         raise NotImplementedError("Command {} is not implemented".format(args.subparser_name))
-+ 
- 
- 
- if __name__ == "__main__":


=====================================
debian/patches/series
=====================================
@@ -1,4 +1,3 @@
 fix_linking.patch
 fix_test.patch
-fix-out-dir.patch
 extend-test-datetime-to-dec-2021.patch


=====================================
debian/watch
=====================================
@@ -1,3 +1,3 @@
 version=4
 
-https://github.com/conda/conda-package-handling/releases .*/archive/v?@ANY_VERSION@@ARCHIVE_EXT@
+https://github.com/conda/conda-package-handling/releases .*/archive/.*/v?@ANY_VERSION@@ARCHIVE_EXT@


=====================================
devtools/conda-envs/test_env.yaml
=====================================
@@ -0,0 +1,4 @@
+name: test_env
+dependencies:
+  - anaconda-client
+  - conda-build


=====================================
src/conda_package_handling/_version.py
=====================================
@@ -23,9 +23,9 @@ def get_keywords():
     # setup.py/versioneer.py will grep for the variable names, so they must
     # each be defined on a line of their own. _version.py will just call
     # get_keywords().
-    git_refnames = " (HEAD -> master, tag: 1.7.2)"
-    git_full = "29b05daaa51ff8c790fa31c401ce372f3a68514a"
-    git_date = "2020-10-16 19:02:10 +0200"
+    git_refnames = " (HEAD -> master, tag: 1.7.3)"
+    git_full = "3a8c263f86b2d8b2095684e5b9b787609b0e1b34"
+    git_date = "2021-04-12 11:49:45 +0200"
     keywords = {"refnames": git_refnames, "full": git_full, "date": git_date}
     return keywords
 


=====================================
src/conda_package_handling/api.py
=====================================
@@ -76,14 +76,21 @@ def create(prefix, file_list, out_fn, out_folder=None, **kw):
         except:
             raise
 
+    out = None
     for ext in SUPPORTED_EXTENSIONS:
         if out_fn.endswith(ext):
             try:
                 out = SUPPORTED_EXTENSIONS[ext].create(prefix, file_list, out_fn, out_folder, **kw)
-            except:
+                break
+            except Exception as err:
                 # don't leave broken files around
-                if _os.path.isfile(out):
+                if out and _os.path.isfile(out):
                     _rm_rf(out)
+                raise err
+    else:
+        raise ValueError("Didn't recognize extension for file '{}'.  Supported extensions are: {}"
+                         .format(fn, list(SUPPORTED_EXTENSIONS.keys())))
+
     return out
 
 


=====================================
src/conda_package_handling/archive_utils_c.c
=====================================
@@ -134,7 +134,7 @@ static int copy_data(struct archive *ar, struct archive *aw)
             return (ARCHIVE_OK);
         if (r < ARCHIVE_OK)
             return (r);
-        r = archive_write_data_block(aw, buff, size, (int)offset);
+        r = archive_write_data_block(aw, buff, size, (size_t)offset);
         if (r < ARCHIVE_OK) {
             return (r);
     }


=====================================
src/conda_package_handling/cli.py
=====================================
@@ -73,7 +73,7 @@ def parse_args(parse_this=None):
 
 def main(args=None):
     args = parse_args(args)
-    if args.out_folder:
+    if hasattr(args, "out_folder") and args.out_folder:
         args.out_folder = os.path.abspath(os.path.normpath(os.path.expanduser(args.out_folder))) + os.sep
     if args.subparser_name in ('extract', 'x'):
         if args.info:


=====================================
src/conda_package_handling/tarball.py
=====================================
@@ -5,6 +5,7 @@ import subprocess
 import sys
 import tarfile
 from tempfile import NamedTemporaryFile
+import logging
 
 try:
     from . import archive_utils
@@ -16,6 +17,7 @@ from . import utils
 from .interface import AbstractBaseFormat
 from .exceptions import CaseInsensitiveFileSystemError, InvalidArchiveError
 
+LOG = logging.getLogger(__file__)
 
 def _sort_file_order(prefix, files):
     """Sort by filesize or by binsort, to optimize compression"""
@@ -34,7 +36,6 @@ def _sort_file_order(prefix, files):
             else:
                 info_order = 1 + abs(hash(ext)) % (10 ** 8)
         return info_order, fsize
-
     binsort = os.path.join(sys.prefix, 'bin', 'binsort')
     if os.path.exists(binsort):
         with NamedTemporaryFile(mode='w', suffix='.filelist', delete=False) as fl:
@@ -49,6 +50,15 @@ def _sort_file_order(prefix, files):
                 files_list = [f.split(prefix + os.sep, 1)[-1]
                                 for f in files_list]
                 os.unlink(fl.name)
+        # Binsort does not handle symlinks gracefully. It will follow them. We must correct that.
+        s1 = set(files)
+        s2 = set(files_list)
+        followed = s2 - s1
+        for f in followed:
+            files_list.remove(f)
+        s2 = set(files_list)
+        if len(s1) > len(s2):
+            files_list.extend(s1 - s2)
     else:
         files_list = list(f for f in sorted(files, key=order))
     return files_list
@@ -132,7 +142,14 @@ class CondaTarBZ2(AbstractBaseFormat):
             fn = os.path.normpath(os.path.join(os.getcwd(), fn))
 
         if libarchive_enabled:
-            _tar_xf(fn, dest_dir)
+            try:
+                _tar_xf(fn, dest_dir)
+            except InvalidArchiveError:
+                LOG.warning(
+                    "Failed extraction with libarchive... falling back to python implementation"
+                )
+                _tar_xf_no_libarchive(fn, dest_dir)
+
         else:
             _tar_xf_no_libarchive(fn, dest_dir)
 


=====================================
tests/test_api.py
=====================================
@@ -188,6 +188,10 @@ def test_secure_refusal_to_extract_abs_paths(testing_workdir):
     with tarfile.open('pinkie.tar.bz2', 'w:bz2') as tf:
         open('thebrain', 'w').close()
         tf.add(os.path.join(testing_workdir, 'thebrain'), '/naughty/abs_path')
+        try:
+            tf.getmember('/naughty/abs_path')
+        except KeyError:
+            pytest.skip("Tar implementation does not generate unsafe paths in archive.")
 
     with pytest.raises(api.InvalidArchiveError):
         api.extract('pinkie.tar.bz2')



View it on GitLab: https://salsa.debian.org/med-team/conda-package-handling/-/compare/753c68206a21e611b0ed046b26ee36422e729154...844e46f103818a6b0225ceb2e8c831976f9c66b9

-- 
View it on GitLab: https://salsa.debian.org/med-team/conda-package-handling/-/compare/753c68206a21e611b0ed046b26ee36422e729154...844e46f103818a6b0225ceb2e8c831976f9c66b9
You're receiving this email because of your account on salsa.debian.org.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/debian-med-commit/attachments/20210420/038919b5/attachment-0001.htm>


More information about the debian-med-commit mailing list