[med-svn] [Git][med-team/python-scitrack][upstream] New upstream version 2020.6.5
Nilesh Patra
gitlab at salsa.debian.org
Mon Jul 13 19:31:16 BST 2020
Nilesh Patra pushed to branch upstream at Debian Med / python-scitrack
Commits:
d6b638cb by Nilesh Patra at 2020-07-13T18:10:36+00:00
New upstream version 2020.6.5
- - - - -
14 changed files:
- + .gitattributes
- + .github/workflows/testing_develop.yml
- .gitignore
- .hgignore
- .hgtags
- + LICENSE
- README.rst
- + pyproject.toml
- setup.py
- scitrack/__init__.py → src/scitrack/__init__.py
- + tests/sample-crlf.fasta
- tests/sample.fasta → tests/sample-lf.fasta
- tests/test_logging.py
- + tox.ini
Changes:
=====================================
.gitattributes
=====================================
@@ -0,0 +1 @@
+tests/*fasta binary
=====================================
.github/workflows/testing_develop.yml
=====================================
@@ -0,0 +1,38 @@
+name: CI
+
+on:
+ push:
+ branches: [ "develop" ]
+ pull_request:
+ branches: [ "develop" ]
+
+jobs:
+ tests:
+ name: "Python ${{ matrix.python-version }}"
+ runs-on: ${{ matrix.os }}
+
+ strategy:
+ matrix:
+ os: [windows-latest, macos-latest, ubuntu-latest]
+ python-version: [3.6, 3.7, 3.8]
+
+ steps:
+ - uses: "actions/checkout at v2"
+ - uses: "actions/setup-python at v1"
+ with:
+ python-version: "${{ matrix.python-version }}"
+ - name: "Install dependencies"
+ run: |
+ python -VV
+ python -m site
+ python -m pip install --upgrade pip setuptools wheel
+ python -m pip install --upgrade tox tox-gh-actions
+
+ - name: "Run tox targets for ${{ matrix.python-version }}"
+ run: "python -m tox"
+
+ - name: Upload py3.8 coverage to Codecov
+ uses: codecov/codecov-action at v1
+ with:
+ file: ./tests/junit-*.xml
+ fail_ci_if_error: true
=====================================
.gitignore
=====================================
@@ -26,6 +26,7 @@ pip-log.txt
.tox
nosetests.xml
tests/draw_results
+*.pytest_cache
# Translations
*.mo
=====================================
.hgignore
=====================================
@@ -21,4 +21,5 @@ dist/*
scitrack.egg*
*.sublime-*
*.wpu
-*.pytest_cache
\ No newline at end of file
+*.pytest_cache
+*.tox
\ No newline at end of file
=====================================
.hgtags
=====================================
@@ -1,3 +1,4 @@
2c80657fecfe617eab8b5e071da8b4b494ca3636 0.1.6
402b7daea661f3904f85f6f248f46c8d9f588704 0.1.7
038183f48645c7ba0417fa98946689f86efca803 0.1.8
+d8aa7076747c991858750c0ecfa5915c6d421d5f 0.1.8.1
=====================================
LICENSE
=====================================
@@ -0,0 +1,11 @@
+Copyright 2019-2020 Gavin Huttley
+
+Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
=====================================
README.rst
=====================================
@@ -1,10 +1,26 @@
+|Build Status| |codecov| |Using Black Formatting| |Python 3.6+|
+
+.. |Build Status| image:: https://github.com/HuttleyLab/scitrack/workflows/CI/badge.svg?branch=develop
+ :target: https://github.com/HuttleyLab/scitrack/actions?workflow=CI
+ :alt: CI Status
+
+.. |codecov| image:: https://codecov.io/gh/HuttleyLab/scitrack/branch/develop/graph/badge.svg
+ :target: https://codecov.io/gh/HuttleyLab/scitrack
+
+.. |Using Black Formatting| image:: https://img.shields.io/badge/code%20style-black-000000.svg
+.. |Python 3.6+| image:: https://img.shields.io/badge/python-3.6+-blue.svg
+ :target: https://www.python.org/downloads/release/python-360/
+
+
##################
About ``scitrack``
##################
-One of the critical challenges in scientific analysis is to track all the elements involved. This includes the arguments provided to a specific application, input data files referenced by those arguments and output data generated by the application. In addition to this, tracking a minimal set of system specific information.
+One of the critical challenges in scientific analysis is to track all the elements involved. This includes the arguments provided to a specific application (including default values), input data files referenced by those arguments and output data generated by the application. In addition to this, tracking a minimal set of system specific information.
+
+``scitrack`` is a simple package aimed at researchers writing scripts, or more substantial scientific software, to support the tracking of scientific computation. The package provides elementary functionality to support logging. The primary capabilities concern generating checksums on input and output files and facilitating logging of the computational environment.
-``scitrack`` is a library aimed at application developers writing scientific software to support this tracking of scientific computation. The library provides elementary functionality to support logging. The primary capabilities concern generating checksums on input and output files and facilitating logging of the computational environment.
+To see some projects using ``scitrack``, see the "Used by" link at the top of the `project GitHub page <https://github.com/HuttleyLab/scitrack>`_..
**********
Installing
@@ -30,14 +46,12 @@ And then install::
``CachingLogger``
*****************
-There is a single object provided by ``scitrack``, ``CachingLogger``. This object is basically a wrapper around the ``logging`` module, but on invocation, captures basic information regarding the system and the command line call that was made to invoke the application.
+There is a single object provided by ``scitrack``, ``CachingLogger``. This object is basically a wrapper around the Python standard library ``logging`` module. On invocation, ``CachingLogger`` captures basic information regarding the system and the command line call that was made to invoke the application.
-In addition, the class provides convenience methods for logging both the path and the md5 hexdigest checksum of input/output files. A method is also provided for producing checksums of text data. The latter is useful for the case when data are from a stream or a database, for instance.
+In addition, the class provides convenience methods for logging both the path and the md5 hexdigest checksum [1]_ of input/output files. A method is also provided for producing checksums of text data. The latter is useful for the case when data are from a stream or a database, for instance.
All logging calls are cached until a path for a logfile is provided. The logger can also, optionally, create directories.
-When run in parallel using ``mpirun``, the process ID is appended to the hostname to help identify processors.
-
**********************************
Simple instantiation of the logger
**********************************
@@ -49,7 +63,6 @@ Creating the logger. Setting ``create_dir=True`` means on creation of the logfil
from scitrack import CachingLogger
LOGGER = CachingLogger(create_dir=True)
LOGGER.log_file_path = "somedir/some_path.log"
-
The last assignment triggers creation of ``somedir/some_path.log``.
@@ -57,40 +70,38 @@ The last assignment triggers creation of ``somedir/some_path.log``.
Capturing a programs arguments and options
******************************************
-``scitrack`` will write the contents of ``sys.argv`` to the log file, prefixed by ``command_string``. However, this only captures arguments specified on the command line. Tracking the value of optional arguments not specified, which may have default values, is critical to tracking the full command set. Doing this is your responsibility as a developer.
+``scitrack`` will write the contents of ``sys.argv`` to the log file, prefixed by ``command_string``. However, this only captures arguments specified on the command line. Tracking the value of optional arguments not specified, which may have default values, is critical to tracking the full command set. Doing this is now easy with the simple statement ``LOGGER.log_args()``. The logger can also record the versions of named dependencies..
-Here's one approach when using the ``click`` `command line interface library <http://click.pocoo.org/>`_. Below we create a simple ``click`` app and capture the required and optional argument values.
+Here's one approach to incorporating ``scitrack`` into a command line application built using the ``click`` `command line interface library <http://click.pocoo.org/>`_. Below we create a simple ``click`` app and capture the required and optional argument values.
.. code:: python
- from scitrack import CachingLogger
import click
+ from scitrack import CachingLogger
+
LOGGER = CachingLogger()
- @click.group()
- def main():
- """the main command"""
- pass
- @main.command()
- @click.option('-i', '--infile', type=click.Path(exists=True))
- @click.option('-t', '--test', is_flag=True, help='Run test.')
- def my_app(infile, test):
+ @click.command()
+ @click.option("-i", "--infile", type=click.Path(exists=True))
+ @click.option("-t", "--test", is_flag=True, help="Run test.")
+ def main(infile, test):
# capture the local variables, at this point just provided arguments
LOGGER.log_args()
- LOGGER.log_versions('numpy')
+ LOGGER.log_versions("numpy")
LOGGER.input_file(infile)
LOGGER.log_file_path = "some_path.log"
+
if __name__ == "__main__":
- my_app()
+ main()
The ``CachingLogger.write()`` method takes a message and a label. All other logging methods wrap ``log_message()``, providing a specific label. For instance, the method ``input_file()`` writes out two lines in the log.
-- input_file_path, the absolute path to the intput file
-- input_file_path md5sum, the hex digest of the file
+- ``input_file_path``, the absolute path to the intput file
+- ``input_file_path md5sum``, the hex digest of the file
``output_file()`` behaves analogously. An additional method ``text_data()`` is useful for other data input/output sources (e.g. records from a database). For this to have value for arbitrary data types requires a systematic approach to ensuring the text conversion is robust across platforms..
@@ -104,17 +115,26 @@ Some sample output
::
- 2018-11-28 11:33:30 yourmachine.com:71779 INFO system_details : system=Darwin Kernel Version 18.2.0: Fri Oct 5 19:41:49 PDT 2018; root:xnu-4903.221.2~2/RELEASE_X86_64
- 2018-11-28 11:33:30 yourmachine.com:71779 INFO python : 3.7.1
- 2018-11-28 11:33:30 yourmachine.com:71779 INFO user : gavin
- 2018-11-28 11:33:30 yourmachine.com:71779 INFO command_string : /Users/gavin/miniconda3/envs/py37/bin/py.test -s
- 2018-11-28 11:33:30 yourmachine.com:71779 INFO input_file_path : /Users/gavin/repos/SciTrack/tests/sample.fasta
- 2018-11-28 11:33:30 yourmachine.com:71779 INFO input_file_path md5sum : 96eb2c2632bae19eb65ea9224aaafdad
- 2018-11-28 11:33:30 yourmachine.com:71779 INFO version : test_logging==0.1.5
- 2018-11-28 11:33:30 yourmachine.com:71779 INFO version : numpy==1.15.1
+ 2020-05-25 13:32:07 Eratosthenes:98447 INFO system_details : system=Darwin Kernel Version 19.4.0: Wed Mar 4 22:28:40 PST 2020; root:xnu-6153.101.6~15/RELEASE_X86_64
+ 2020-05-25 13:32:07 Eratosthenes:98447 INFO python : 3.8.2
+ 2020-05-25 13:32:07 Eratosthenes:98447 INFO user : gavin
+ 2020-05-25 13:32:07 Eratosthenes:98447 INFO command_string : ./demo.py -i /Users/gavin/repos/SciTrack/tests/sample-lf.fasta
+ 2020-05-25 13:32:07 Eratosthenes:98447 INFO params : {'infile': '/Users/gavin/repos/SciTrack/tests/sample-lf.fasta', 'test': False}
+ 2020-05-25 13:32:07 Eratosthenes:98447 INFO version : __main__==None
+ 2020-05-25 13:32:07 Eratosthenes:98447 INFO version : numpy==1.18.4
+ 2020-05-25 13:32:07 Eratosthenes:98447 INFO input_file_path : /Users/gavin/repos/SciTrack/tests/sample-lf.fasta
+ 2020-05-25 13:32:07 Eratosthenes:98447 INFO input_file_path md5sum : 96eb2c2632bae19eb65ea9224aaafdad
**********************
Other useful functions
**********************
-Two other useful functions are ``get_file_hexdigest`` and ``get_text_hexdigest``. The latter can take either unicode or ascii strings.
+Two other useful functions are ``get_file_hexdigest`` and ``get_text_hexdigest``.
+
+****************
+Reporting issues
+****************
+
+Use the project `issue tracker <https://github.com/HuttleyLab/scitrack/issues>`_.
+
+.. [1] The hexdigest serves as a unique signature of a files contents.
=====================================
pyproject.toml
=====================================
@@ -0,0 +1,29 @@
+[build-system]
+requires = ["setuptools>=40.6.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[tool.black]
+line-length = 88
+exclude = '''
+/(
+ \.eggs
+ | \.git
+ | \.hg
+ | \.mypy_cache
+ | \.tox
+ | \.venv
+ | _build
+ | build
+ | dist
+ | tests/data
+)/
+'''
+
+[tool.isort]
+atomic=true
+force_grid_wrap=0
+include_trailing_comma=true
+lines_after_imports=2
+lines_between_types=1
+multi_line_output=3
+use_parentheses=true
=====================================
setup.py
=====================================
@@ -1,20 +1,30 @@
#!/usr/bin/env python
-from setuptools import setup
-import sys
import pathlib
+import sys
+
+from setuptools import find_packages, setup
+
__author__ = "Gavin Huttley"
-__copyright__ = "Copyright 2016, Gavin Huttley"
+__copyright__ = "Copyright 2016-2020, Gavin Huttley"
__credits__ = ["Gavin Huttley"]
__license__ = "BSD"
-__version__ = "0.1.8.1"
+__version__ = "2020.6.5"
__maintainer__ = "Gavin Huttley"
__email__ = "Gavin.Huttley at anu.edu.au"
__status__ = "Development"
if sys.version_info < (3, 6):
py_version = ".".join([str(n) for n in sys.version_info])
- raise RuntimeError("Python-3.6 or greater is required, Python-%s used." % py_version)
+ raise RuntimeError(
+ "Python-3.6 or greater is required, Python-%s used." % py_version
+ )
+
+PROJECT_URLS = {
+ "Documentation": "https://github.com/HuttleyLab/scitrack",
+ "Bug Tracker": "https://github.com/HuttleyLab/scitrack/issues",
+ "Source Code": "https://github.com/HuttleyLab/scitrack",
+}
short_description = "scitrack"
@@ -22,6 +32,8 @@ readme_path = pathlib.Path(__file__).parent / "README.rst"
long_description = readme_path.read_text()
+PACKAGE_DIR = "src"
+
setup(
name="scitrack",
version=__version__,
@@ -29,18 +41,23 @@ setup(
author_email="gavin.huttley at anu.edu.au",
description=short_description,
long_description=long_description,
+ long_description_content_type="text/x-rst",
platforms=["any"],
license=[__license__],
- keywords=["science", "logging", "parallel"],
+ keywords=["science", "logging"],
classifiers=[
- "Development Status :: 3 - Alpha",
- "Intended Audience :: Science/Research",
- "Programming Language :: Python :: 3",
- "License :: OSI Approved :: BSD License",
- "Topic :: Scientific/Engineering :: Bio-Informatics",
- "Topic :: Software Development :: Libraries :: Python Modules",
- "Operating System :: OS Independent",
- ],
- packages=["scitrack"],
+ "Development Status :: 4 - Beta",
+ "Intended Audience :: Science/Research",
+ "License :: OSI Approved :: BSD License",
+ "Topic :: Scientific/Engineering :: Bio-Informatics",
+ "Topic :: Software Development :: Libraries :: Python Modules",
+ "Operating System :: OS Independent",
+ "Programming Language :: Python :: 3.6",
+ "Programming Language :: Python :: 3.7",
+ "Programming Language :: Python :: 3.8",
+ ],
+ packages=find_packages(where="src"),
+ package_dir={"": PACKAGE_DIR},
url="https://github.com/HuttleyLab/scitrack",
- )
+ project_urls=PROJECT_URLS,
+)
=====================================
scitrack/__init__.py → src/scitrack/__init__.py
=====================================
@@ -6,13 +6,15 @@ import os
import platform
import socket
import sys
+
from getpass import getuser
+
__author__ = "Gavin Huttley"
-__copyright__ = "Copyright 2016, Gavin Huttley"
+__copyright__ = "Copyright 2016-2020, Gavin Huttley"
__credits__ = ["Gavin Huttley"]
__license__ = "BSD"
-__version__ = "0.1.8.1"
+__version__ = "2020.6.5"
__maintainer__ = "Gavin Huttley"
__email__ = "Gavin.Huttley at anu.edu.au"
__status__ = "Development"
@@ -93,14 +95,16 @@ class CachingLogger(object):
self.log_file_path = log_file_path
def _reset(self, mode="w"):
- self._log_file_path = None
self._mode = mode
self._started = False
self._messages = []
if self._logfile is not None:
+ self._logfile.flush()
self._logfile.close()
self._logfile = None
+ self._log_file_path = None
+
@property
def log_file_path(self):
return self._log_file_path
@@ -196,9 +200,6 @@ class CachingLogger(object):
def shutdown(self):
"""safely shutdown the logger"""
logging.getLogger().removeHandler(self._logfile)
- self._logfile.flush()
- self._logfile.close()
- self._logfile = None
self._reset()
def log_versions(self, packages=None):
@@ -280,9 +281,12 @@ def get_text_hexdigest(data):
This will occur if the line ending character differs from being read in
'rb' versus 'r' modes.
"""
- if data.__class__ not in ("".__class__, u"".__class__):
- raise TypeError("can only checksum string or unicode data")
- data = data.encode("utf-8")
+ data_class = data.__class__
+ if data_class in ("".__class__, u"".__class__):
+ data = data.encode("utf-8")
+ elif data.__class__ != b"".__class__:
+ raise TypeError("can only checksum string, unicode or bytes data")
+
md5 = hashlib.md5()
md5.update(data)
return md5.hexdigest()
=====================================
tests/sample-crlf.fasta
=====================================
Binary files /dev/null and b/tests/sample-crlf.fasta differ
=====================================
tests/sample.fasta → tests/sample-lf.fasta
=====================================
=====================================
tests/test_logging.py
=====================================
@@ -1,35 +1,46 @@
# -*- coding: utf-8 -*-
-
-import os
import shutil
-import subprocess
import sys
+
from collections import Counter
+from pathlib import Path
+from tempfile import TemporaryDirectory
+
+import pytest
+
+from scitrack import (
+ CachingLogger,
+ get_file_hexdigest,
+ get_package_name,
+ get_text_hexdigest,
+ get_version_for_package,
+)
-from scitrack import (CachingLogger, get_package_name, get_text_hexdigest,
- get_version_for_package, logging)
__author__ = "Gavin Huttley"
-__copyright__ = "Copyright 2016, Gavin Huttley"
+__copyright__ = "Copyright 2016-2020, Gavin Huttley"
__credits__ = ["Gavin Huttley"]
__license__ = "BSD"
-__version__ = "0.1.8.1"
+__version__ = "2020.6.5"
__maintainer__ = "Gavin Huttley"
__email__ = "Gavin.Huttley at anu.edu.au"
__status__ = "Development"
-LOGFILE_NAME = "delme.log"
-DIRNAME = "delme"
+TEST_ROOTDIR = Path(__file__).parent
+
+DIRNAME = TEST_ROOTDIR / "delme"
+LOGFILE_NAME = DIRNAME / "delme.log"
def test_creates_path():
"""creates a log path"""
LOGGER = CachingLogger(create_dir=True)
- LOGGER.log_file_path = os.path.join(DIRNAME, LOGFILE_NAME)
- LOGGER.input_file("sample.fasta")
+ LOGGER.log_file_path = LOGFILE_NAME
+ LOGGER.input_file(TEST_ROOTDIR / "sample-lf.fasta")
LOGGER.shutdown()
- assert os.path.exists(DIRNAME)
- assert os.path.exists(os.path.join(DIRNAME, LOGFILE_NAME))
+ assert DIRNAME.exists()
+ assert LOGFILE_NAME.exists()
+
try:
shutil.rmtree(DIRNAME)
except OSError:
@@ -39,15 +50,18 @@ def test_creates_path():
def test_tracks_args():
"""details on host, python version should be present in log"""
LOGGER = CachingLogger(create_dir=True)
- LOGGER.log_file_path = os.path.join(LOGFILE_NAME)
- LOGGER.input_file("sample.fasta")
+ LOGGER.log_file_path = LOGFILE_NAME
+ LOGGER.input_file(TEST_ROOTDIR / "sample-lf.fasta")
LOGGER.shutdown()
- with open(LOGFILE_NAME, "r") as infile:
- contents = "".join(infile.readlines())
- for label in ["system_details", "python", "user", "command_string"]:
- assert contents.count(label) == 1, (label, contents.count(label))
+ contents = LOGFILE_NAME.read_text()
+ for label in ["system_details", "python", "user", "command_string"]:
+ assert contents.count(f"\t{label}") == 1, (
+ label,
+ contents.count(label),
+ )
+
try:
- os.remove(LOGFILE_NAME)
+ shutil.rmtree(DIRNAME)
except OSError:
pass
@@ -55,7 +69,7 @@ def test_tracks_args():
def test_tracks_locals():
"""details on local arguments should be present in log"""
LOGGER = CachingLogger(create_dir=True)
- LOGGER.log_file_path = os.path.join(LOGFILE_NAME)
+ LOGGER.log_file_path = LOGFILE_NAME
def track_func(a=1, b="abc"):
LOGGER.log_args()
@@ -69,9 +83,9 @@ def test_tracks_locals():
got = eval(line.split("params :")[1])
break
assert got == dict(a=1, b="abc")
+
try:
- os.remove(LOGFILE_NAME)
- pass
+ shutil.rmtree(DIRNAME)
except OSError:
pass
@@ -100,23 +114,78 @@ def test_package_versioning():
def test_tracks_versions():
"""should track versions"""
LOGGER = CachingLogger(create_dir=True)
- LOGGER.log_file_path = os.path.join(LOGFILE_NAME)
- LOGGER.input_file("sample.fasta")
+ LOGGER.log_file_path = LOGFILE_NAME
+ LOGGER.input_file(TEST_ROOTDIR / "sample-lf.fasta")
LOGGER.log_versions(["numpy"])
LOGGER.shutdown()
- with open(LOGFILE_NAME, "r") as infile:
- contents = "".join(infile.readlines())
- for label in ["system_details", "python", "user", "command_string"]:
- assert contents.count(label) == 1, (label, contents.count(label))
- for line in contents.splitlines():
- if "version :" in line:
- if "numpy" not in line:
- assert "==%s" % __version__ in line, line
- else:
- assert "numpy" in line, line
- print("\n\n", contents)
+ contents = LOGFILE_NAME.read_text()
+ for label in ["system_details", "python", "user", "command_string"]:
+ assert contents.count(f"\t{label}") == 1, (
+ label,
+ contents.count(label),
+ )
+ for line in contents.splitlines():
+ if "version :" in line:
+ if "numpy" not in line:
+ assert "==%s" % __version__ in line, line
+ else:
+ assert "numpy" in line, line
+
+ try:
+ shutil.rmtree(DIRNAME)
+ except OSError:
+ pass
+
+
+def test_caching():
+ """should cache calls prior to logging"""
+ LOGGER = CachingLogger(create_dir=True)
+ LOGGER.input_file(TEST_ROOTDIR / "sample-lf.fasta")
+ assert (
+ "sample-lf.fasta" in LOGGER._messages[-2] and "md5sum" in LOGGER._messages[-1]
+ )
+ LOGGER.log_versions(["numpy"])
+ assert "numpy==" in LOGGER._messages[-1]
+
+ LOGGER.log_file_path = LOGFILE_NAME
+ LOGGER.shutdown()
+ try:
+ shutil.rmtree(DIRNAME)
+ except OSError:
+ pass
+
+
+def test_shutdown():
+ """correctly purges contents"""
+ LOGGER = CachingLogger(create_dir=True)
+ LOGGER.log_file_path = LOGFILE_NAME
+ LOGGER.input_file(TEST_ROOTDIR / "sample-lf.fasta")
+ LOGGER.shutdown()
+ try:
+ shutil.rmtree(DIRNAME)
+ except OSError:
+ pass
+
+
+def test_tracks_versions_empty():
+ """should track version of scitrack"""
+ LOGGER = CachingLogger(create_dir=True)
+ LOGGER.log_file_path = LOGFILE_NAME
+ LOGGER.input_file(TEST_ROOTDIR / "sample-lf.fasta")
+ LOGGER.log_versions()
+ LOGGER.shutdown()
+ contents = LOGFILE_NAME.read_text()
+ for label in ["system_details", "python", "user", "command_string"]:
+ assert contents.count(f"\t{label}") == 1, (
+ label,
+ contents.count(label),
+ )
+ for line in contents.splitlines():
+ if "version :" in line:
+ assert "==%s" % __version__ in line, line
+
try:
- os.remove(LOGFILE_NAME)
+ shutil.rmtree(DIRNAME)
except OSError:
pass
@@ -124,7 +193,7 @@ def test_tracks_versions():
def test_tracks_versions_string():
"""should track version if package name is a string"""
LOGGER = CachingLogger(create_dir=True)
- LOGGER.log_file_path = os.path.join(LOGFILE_NAME)
+ LOGGER.log_file_path = LOGFILE_NAME
LOGGER.log_versions("numpy")
LOGGER.shutdown()
import numpy
@@ -136,16 +205,42 @@ def test_tracks_versions_string():
for line in contents.splitlines():
if "version :" in line and "numpy" in line:
assert expect in line, line
+
try:
- os.remove(LOGFILE_NAME)
+ shutil.rmtree(DIRNAME)
except OSError:
pass
+def test_get_version_for_package():
+ """should track version if package is a module"""
+ import numpy
+
+ got = get_version_for_package(numpy)
+ assert got == numpy.__version__
+ # one with a callable
+ pyfile = TEST_ROOTDIR / "delme.py"
+ pyfile.write_text("\n".join(["def version():", " return 'my-version'"]))
+ sys.path.append(TEST_ROOTDIR)
+ import delme
+
+ got = get_version_for_package("delme")
+ assert got == "my-version"
+ pyfile.unlink()
+
+ # func returns a list
+ pyfile.write_text("version = ['my-version']\n")
+ from importlib import reload
+
+ got = get_version_for_package(reload(delme))
+ assert got == "my-version"
+ pyfile.unlink()
+
+
def test_tracks_versions_module():
"""should track version if package is a module"""
LOGGER = CachingLogger(create_dir=True)
- LOGGER.log_file_path = os.path.join(LOGFILE_NAME)
+ LOGGER.log_file_path = LOGFILE_NAME
import numpy
expect = "numpy==%s" % numpy.__version__
@@ -157,8 +252,9 @@ def test_tracks_versions_module():
for line in contents.splitlines():
if "version :" in line and "numpy" in line:
assert expect in line, line
+
try:
- os.remove(LOGFILE_NAME)
+ shutil.rmtree(DIRNAME)
except OSError:
pass
@@ -167,7 +263,7 @@ def test_appending():
"""appending to an existing logfile should work"""
LOGGER = CachingLogger(create_dir=True)
LOGGER.log_file_path = LOGFILE_NAME
- LOGGER.input_file("sample.fasta")
+ LOGGER.input_file(TEST_ROOTDIR / "sample-lf.fasta")
LOGGER.shutdown()
records = Counter()
with open(LOGFILE_NAME) as infile:
@@ -178,7 +274,7 @@ def test_appending():
LOGGER = CachingLogger(create_dir=True)
LOGGER.mode = "a"
LOGGER.log_file_path = LOGFILE_NAME
- LOGGER.input_file("sample.fasta")
+ LOGGER.input_file(TEST_ROOTDIR / "sample-lf.fasta")
LOGGER.shutdown()
records = Counter()
@@ -190,7 +286,7 @@ def test_appending():
assert vals == {2}
try:
- os.remove(LOGFILE_NAME)
+ shutil.rmtree(DIRNAME)
except OSError:
pass
@@ -198,30 +294,137 @@ def test_appending():
def test_mdsum_input():
"""md5 sum of input file should be correct"""
LOGGER = CachingLogger(create_dir=True)
- LOGGER.log_file_path = os.path.join(LOGFILE_NAME)
- LOGGER.input_file("sample.fasta")
+ LOGGER.log_file_path = LOGFILE_NAME
+ # first file has LF, second has CRLF line endings
+ hex_path = [
+ ("96eb2c2632bae19eb65ea9224aaafdad", "sample-lf.fasta"),
+ ("e7e219f66be15d8afc7cdb85303305a7", "sample-crlf.fasta"),
+ ]
+ LOGGER.input_file(TEST_ROOTDIR / "sample-lf.fasta")
+ LOGGER.input_file(TEST_ROOTDIR / "sample-crlf.fasta")
LOGGER.shutdown()
with open(LOGFILE_NAME, "r") as infile:
num = 0
for line in infile:
- line = line.strip()
- if "input_file_path md5sum" in line:
- assert "96eb2c2632bae19eb65ea9224aaafdad" in line
- num += 1
- assert num == 1
+ for h, p in hex_path:
+ if p in line:
+ assert "input_file" in line
+ line = next(infile)
+ assert h in line
+ num += 1
+
+ assert num == len(hex_path)
try:
- os.remove(LOGFILE_NAME)
+ shutil.rmtree(DIRNAME)
+ except OSError:
+ pass
+
+
+def test_mdsum_output():
+ """md5 sum of output file should be correct"""
+ LOGGER = CachingLogger(create_dir=True)
+ LOGGER.log_file_path = LOGFILE_NAME
+ # first file has LF, second has CRLF line endings
+ hex_path = [
+ ("96eb2c2632bae19eb65ea9224aaafdad", "sample-lf.fasta"),
+ ]
+ LOGGER.output_file(TEST_ROOTDIR / "sample-lf.fasta")
+ LOGGER.shutdown()
+
+ with open(LOGFILE_NAME, "r") as infile:
+ num = 0
+ for line in infile:
+ for h, p in hex_path:
+ if p in line:
+ line = next(infile)
+ assert h in line
+ num += 1
+
+ assert num == len(hex_path)
+
+ try:
+ shutil.rmtree(DIRNAME)
+ except OSError:
+ pass
+
+
+def test_logging_text():
+ """correctly logs text data"""
+ text = "abcde\nedfgu\nyhbnd"
+ hexd = "f06597f8a983dfc93744192b505a8af9"
+ LOGGER = CachingLogger(create_dir=True)
+ LOGGER.log_file_path = LOGFILE_NAME
+ LOGGER.text_data(text, label="UNIQUE")
+ LOGGER.shutdown()
+ contents = LOGFILE_NAME.read_text().splitlines()
+ unique = None
+ for line in contents:
+ if "UNIQUE" in line:
+ unique = line
+ break
+ assert hexd in unique
+ try:
+ shutil.rmtree(DIRNAME)
+ except OSError:
+ pass
+
+
+def test_logfile_path():
+ """correctly assigned"""
+ LOGGER = CachingLogger(create_dir=True, log_file_path=LOGFILE_NAME)
+ assert LOGGER.log_file_path == str(LOGFILE_NAME)
+ LOGGER.shutdown()
+ try:
+ shutil.rmtree(DIRNAME)
except OSError:
pass
def test_md5sum_text():
"""md5 sum for text data should be computed"""
- data = u"åbcde"
+ data = "åbcde"
s = get_text_hexdigest(data)
assert s
data = "abcde"
s = get_text_hexdigest(data)
assert s
+
+ # loading contents from files with diff line-endings and check
+ hex_path = [
+ ("96eb2c2632bae19eb65ea9224aaafdad", "sample-lf.fasta"),
+ ("e7e219f66be15d8afc7cdb85303305a7", "sample-crlf.fasta"),
+ ]
+ for h, p in hex_path:
+ p = TEST_ROOTDIR / p
+ data = p.read_bytes()
+ print(p, repr(data))
+ got = get_text_hexdigest(data)
+ assert got == h, (p, repr(data))
+
+
+def test_get_text_hexdigest_invalid():
+ """raises TypeError when invalid data provided"""
+ with pytest.raises(TypeError):
+ get_text_hexdigest(None)
+
+ with pytest.raises(TypeError):
+ get_text_hexdigest([])
+
+
+def test_read_from_written():
+ """create files with different line endings dynamically"""
+ text = "abcdeENDedfguENDyhbnd"
+ with TemporaryDirectory(dir=TEST_ROOTDIR) as dirname:
+ for ex, lf in (
+ ("f06597f8a983dfc93744192b505a8af9", "\n"),
+ ("39db5cc2f7749f02e0c712a3ece12ffc", "\r\n"),
+ ):
+ p = Path(dirname) / "test.txt"
+ data = text.replace("END", lf)
+ p.write_bytes(data.encode("utf-8"))
+ expect = get_text_hexdigest(data)
+ assert expect == ex, (expect, ex)
+ got = get_file_hexdigest(p)
+ assert got == expect, f"FAILED: {repr(lf)}, {(ex, got)}"
=====================================
tox.ini
=====================================
@@ -0,0 +1,33 @@
+[tox]
+envlist = py36, py37, py38
+
+[testenv]
+passenv = *
+deps = numpy
+ click
+ pytest
+ pytest-cov
+
+[testenv:py38]
+changedir = tests
+basepython = python3.8
+commands =
+ pytest --junitxml=junit-{envname}.xml --cov-report xml --cov=scitrack
+
+[testenv:py37]
+changedir = tests
+basepython = python3.7
+commands =
+ pytest --junitxml=junit-{envname}.xml --cov-report xml --cov=scitrack
+
+[testenv:py36]
+changedir = tests
+basepython = python3.6
+commands =
+ pytest --junitxml=junit-{envname}.xml --cov-report xml --cov=scitrack
+
+[gh-actions]
+python =
+ 3.6: py36
+ 3.7: py37
+ 3.8: py38
View it on GitLab: https://salsa.debian.org/med-team/python-scitrack/-/commit/d6b638cb2bed1131fe63e61a81a0446d40896cfc
--
View it on GitLab: https://salsa.debian.org/med-team/python-scitrack/-/commit/d6b638cb2bed1131fe63e61a81a0446d40896cfc
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/debian-med-commit/attachments/20200713/a264d8ac/attachment-0001.html>
More information about the debian-med-commit
mailing list