[med-svn] [debian-med-benchmarking-spec.git] 02/02: Presented version w/ Piotr's edits

[Kevin Murray]

This is an automated email from the git hooks/post-receive script.

daube-guest pushed a commit to branch master
in repository debian-med-benchmarking-spec.git.

commit fca8f8e5d5b1f24fe111be86c3857ffe76da4c50
Author: Kevin Murray <spam at kdmurray.id.au>
Date:   Fri Feb 5 17:17:01 2016 +0100

    Presented version w/ Piotr's edits
---
 benchmarking.md | 48 +++++++++++++++++++++++++++---------------------
 1 file changed, 27 insertions(+), 21 deletions(-)

diff --git a/benchmarking.md b/benchmarking.md
index 28f8f1b..a1addc3 100644
--- a/benchmarking.md
+++ b/benchmarking.md
@@ -26,6 +26,33 @@ for tool's scientific accuracy.
   - Report metrics to Debian infrastructure (UDD) and wider community (SEQWiki,
     bio.tools)
 
+Packages to be tested
+---------------------
+
+All packages within debian med that:
+
+  - Have EDAM-compatible DebTags
+  - Have a CWL tool description for each tool in the package
+    - Should contain EDAM tags per operation
+    - Potentially one CWL tool file per subtool/operation (e.g. `samtools view`
+      vs `samtools sort`)
+  - Are in `main`
+
+Operation
+---------
+
+  - Service that runs benchmarks when:
+    - There is a new version of the underlying package in Debian med unstable
+      - Including any conversion utilities
+    - There is a change in an applicable script for the calculation of metrics
+    - There is a change in an applicable benchmark dataset
+    - There is an applicable transition in progress??
+      - Could catch subtle bugs e.g. py3.4 -> py3.5 issues
+  - Benchmark datasets and scripts for calculation of metrics are automatically
+    selected from repository/DB based on their and the tool's EDAM tags
+  - For reproducibility/debugging, a Docker image performing all setup and
+    preconversion is published along with the benchmark results
+
 
 Architecture:
 -------------
@@ -80,27 +107,6 @@ Architecture:
         be used for the test execution
   - Could all this run on a new instance of debci? or Jenkins?
 
-Requirements
-------------
-
-  - Have an EDAM-compatible DebTags
-  - Have a CWL tool description for each tool in the package
-    - Should contain EDAM tags per operation
-    - Potentially one CWL tool file per subtool/operation (e.g. `samtools view`
-      vs `samtools sort`)
-  - Be in `main`
-
-Operation
----------
-
-  - New service that runs benchmarks when:
-    - There is a new version of the underlying package in Debian med unstable
-      - Including any conversion utilities
-    - There is a change in an applicable script for the calculation of metrics
-    - There is a change in an applicable benchmark dataset
-    - There is an applicable transition in progress??
-      - Could catch subtle bugs e.g. py3.4 -> py3.5 issues
-
 Schema of ideas
 ---------------
 

-- 
Alioth's /usr/local/bin/git-commit-notice on /srv/git.debian.org/git/debian-med/debian-med-benchmarking-spec.git.git