[med-svn] r12534 - in trunk/packages/qiime/trunk/debian: . patches scripts scripts/shell
Timothy Booth
tbooth-guest at alioth.debian.org
Wed Nov 14 13:55:17 UTC 2012
Author: tbooth-guest
Date: 2012-11-14 13:55:16 +0000 (Wed, 14 Nov 2012)
New Revision: 12534
Added:
trunk/packages/qiime/trunk/debian/patches/check_config_file_in_new_location.patch
trunk/packages/qiime/trunk/debian/patches/fix_path_for_support_files
trunk/packages/qiime/trunk/debian/patches/fix_t_test_bug
trunk/packages/qiime/trunk/debian/patches/make_qiime_accept_new_rdp_classifier
Modified:
trunk/packages/qiime/trunk/debian/changelog
trunk/packages/qiime/trunk/debian/control
trunk/packages/qiime/trunk/debian/patches/series
trunk/packages/qiime/trunk/debian/scripts/qiime
trunk/packages/qiime/trunk/debian/scripts/shell/qiime_environment
Log:
Added some patches to address QIIME bugs. The patches are tested but not
pretty, especially the make_qiime_accept_new_rdp_classifier patch which is a
hatchet job.
Modified: trunk/packages/qiime/trunk/debian/changelog
===================================================================
--- trunk/packages/qiime/trunk/debian/changelog 2012-11-14 10:09:03 UTC (rev 12533)
+++ trunk/packages/qiime/trunk/debian/changelog 2012-11-14 13:55:16 UTC (rev 12534)
@@ -1,3 +1,15 @@
+qiime (1.5.0+repack-2ubuntu10) precise; urgency=low
+
+ * Made QIIME accept RDP classifier >2.2 though the patch is
+ a total hack. Solves issue reported by MJC.
+ * Add patch to make QIIME look in the Debian standard location
+ for king.jar and other support_files
+ * Added patch to fix alpha diversity bug and tested by running
+ test_compare_alpha_diversities.py
+ * QIIME now needs python-cogent 1.5.3
+
+ -- Tim Booth <tbooth at ceh.ac.uk> Wed, 14 Nov 2012 09:59:24 +0000
+
qiime (1.5.0+dfsg-1) UNRELEASED; urgency=low
* debian/copyright: - Add Files-Excluded to document what was removed
@@ -6,8 +18,14 @@
-- Andreas Tille <tille at debian.org> Wed, 29 Aug 2012 14:30:02 +0200
-qiime (1.5.0-2ubuntu4) precise; urgency=low
+qiime (1.5.0+repack-2ubuntu5) precise; urgency=low
+ * Tweaked startup script to report the QIIME version
+
+ -- Tim Booth <tbooth at ceh.ac.uk> Thu, 11 Oct 2012 13:49:35 +0100
+
+qiime (1.5.0+repack-2ubuntu4) precise; urgency=low
+
* Remove python26_trim_sff_primers.patch as Qiime now requires
2.7 anyway
* Avoid calling dh_auto_clean at all as it triggers a partial
@@ -19,7 +37,7 @@
* Added patch to self-test where it looks for default config file
* Modified to work with new RDP-classifier package
- -- Tim Booth <tbooth at ceh.ac.uk> Tue, 14 Aug 2012 17:33:26 +0100
+ -- Tim Booth <tbooth at ceh.ac.uk> Tue, 14 Aug 2012 17:53:13 +0100
qiime (1.5.0-2) unstable; urgency=low
Modified: trunk/packages/qiime/trunk/debian/control
===================================================================
--- trunk/packages/qiime/trunk/debian/control 2012-11-14 10:09:03 UTC (rev 12533)
+++ trunk/packages/qiime/trunk/debian/control 2012-11-14 13:55:16 UTC (rev 12534)
@@ -5,7 +5,7 @@
Uploaders: Steffen Moeller <moeller at debian.org>, Tim Booth <tbooth at ceh.ac.uk>,
Andreas Tille <tille at debian.org>
DM-Upload-Allowed: yes
-Build-Depends: debhelper (>= 9), python-all-dev (>= 2.7), python-cogent ( >= 1.5.1 ),
+Build-Depends: debhelper (>= 9), python-all-dev (>= 2.7), python-cogent ( >= 1.5.3 ),
python-numpy, python-matplotlib, ghc6, python-sphinx
Standards-Version: 3.9.3
Homepage: http://www.qiime.org/
@@ -16,8 +16,7 @@
Package: qiime
Architecture: any
Depends: ${shlibs:Depends}, ${misc:Depends}, ${python:Depends},
- python-pynast (>= 1.1)|pynast (>= 1.1), python-cogent ( >= 1.5.1 ),
- king, python-biom-format
+ python-pynast (>= 1.1)|pynast (>= 1.1), python-cogent ( >= 1.5.3 ), king, python-biom-format
Recommends: blast2 | blast+-legacy, cd-hit, rdp-classifier, chimeraslayer, muscle, infernal,
fasttree, ampliconnoise, python-matplotlib, python-numpy, libjs-jquery
Suggests: t-coffee, cytoscape
Added: trunk/packages/qiime/trunk/debian/patches/check_config_file_in_new_location.patch
===================================================================
--- trunk/packages/qiime/trunk/debian/patches/check_config_file_in_new_location.patch (rev 0)
+++ trunk/packages/qiime/trunk/debian/patches/check_config_file_in_new_location.patch 2012-11-14 13:55:16 UTC (rev 12534)
@@ -0,0 +1,15 @@
+We've moved the default config file, so tell this script to look in the new
+location. See the next patch for other path fixes related to support_files
+
+--- a/scripts/print_qiime_config.py
++++ b/scripts/print_qiime_config.py
+@@ -247,8 +247,7 @@
+ """local qiime_config has no extra params"""
+
+ qiime_project_dir = get_qiime_project_dir()
+- orig_config = parse_qiime_config_file(open(qiime_project_dir +
+- '/qiime/support_files/qiime_config'))
++ orig_config = parse_qiime_config_file(open('/etc/qiime/qiime_config'))
+
+ #check the env qiime_config
+ qiime_config_env_filepath = getenv('QIIME_CONFIG_FP')
Added: trunk/packages/qiime/trunk/debian/patches/fix_path_for_support_files
===================================================================
--- trunk/packages/qiime/trunk/debian/patches/fix_path_for_support_files (rev 0)
+++ trunk/packages/qiime/trunk/debian/patches/fix_path_for_support_files 2012-11-14 13:55:16 UTC (rev 12534)
@@ -0,0 +1,88 @@
+Get QIIME to look in the right directory for support_files.
+The default location is relative to the script which is
+in a different location depending on the python version.
+Author: Tim Booth <tbooth at ceh.ac.uk>
+
+---
+The information above should follow the Patch Tagging Guidelines, please
+checkout http://dep.debian.net/deps/dep3/ to learn about the format. Here
+are templates for supplementary fields that you might want to add:
+
+Origin: <vendor|upstream|other>, <url of original patch>
+Bug: <url in upstream bugtracker>
+Bug-Debian: http://bugs.debian.org/<bugnumber>
+Bug-Ubuntu: https://launchpad.net/bugs/<bugnumber>
+Forwarded: <no|not-needed|url proving that it has been forwarded>
+Reviewed-By: <name and email of someone who approved the patch>
+Last-Update: <YYYY-MM-DD>
+
+--- qiime-1.5.0+repack.orig/scripts/plot_taxa_summary.py
++++ qiime-1.5.0+repack/scripts/plot_taxa_summary.py
+@@ -219,8 +219,7 @@ def main():
+ pass
+
+ # move javascript file to javascript output directory
+- shutil.copyfile(os.path.join(qiime_dir,'qiime','support_files',\
+- 'js/overlib.js'),\
++ shutil.copyfile('/usr/lib/qiime/support_files/js/overlib.js',\
+ os.path.join(javascript_path,'overlib.js'))
+
+ #make css output directory
+@@ -231,8 +230,7 @@ def main():
+ pass
+
+ # move css file to css output directory
+- shutil.copyfile(os.path.join(qiime_dir,'qiime','support_files',\
+- 'css/qiime_style.css'),\
++ shutil.copyfile('/usr/lib/qiime/support_files/css/qiime_style.css',\
+ os.path.join(css_path,'qiime_style.css'))
+
+ # verify all parameters are valid
+--- qiime-1.5.0+repack.orig/scripts/make_otu_heatmap_html.py
++++ qiime-1.5.0+repack/scripts/make_otu_heatmap_html.py
+@@ -137,7 +137,8 @@ def main():
+
+ qiime_dir=get_qiime_project_dir()
+
+- js_path=os.path.join(qiime_dir,'qiime/support_files/js')
++ #js_path=os.path.join(qiime_dir,'qiime/support_files/js')
++ js_path='/usr/lib/qiime/support_files/js'
+
+ shutil.copyfile(os.path.join(js_path,'overlib.js'), os.path.join(js_dir_path,'overlib.js'))
+ shutil.copyfile(os.path.join(js_path,'otu_count_display.js'), os.path.join(js_dir_path,'otu_count_display.js'))
+--- qiime-1.5.0+repack.orig/scripts/compare_3d_plots.py
++++ qiime-1.5.0+repack/scripts/compare_3d_plots.py
+@@ -143,7 +143,8 @@ def main():
+
+ qiime_dir=get_qiime_project_dir()
+
+- jar_path=os.path.join(qiime_dir,'qiime/support_files/jar/')
++ #jar_path=os.path.join(qiime_dir,'qiime/support_files/jar/')
++ jar_path='/usr/lib/qiime/support_files/jar/'
+
+ data_dir_path = get_random_directory_name(output_dir=dir_path,
+ return_absolute_path=False)
+--- qiime-1.5.0+repack.orig/scripts/make_distance_histograms.py
++++ qiime-1.5.0+repack/scripts/make_distance_histograms.py
+@@ -142,7 +142,8 @@ def main():
+ for color_info in groups_and_colors:
+ field_to_colors[color_info[0]]=color_info[1:]
+
+- qiime_dir = get_qiime_project_dir()+'/qiime/support_files/'
++ #qiime_dir = get_qiime_project_dir()+'/qiime/support_files/'
++ qiime_dir='/usr/lib/qiime/support_files/'
+
+ fields = opts.fields
+ if fields is not None:
+--- qiime-1.5.0+repack.orig/scripts/make_2d_plots.py
++++ qiime-1.5.0+repack/scripts/make_2d_plots.py
+@@ -148,7 +148,8 @@ def main():
+
+ qiime_dir=get_qiime_project_dir()
+
+- js_path=os.path.join(qiime_dir,'qiime','support_files','js')
++ #js_path=os.path.join(qiime_dir,'qiime','support_files','js')
++ js_path='/usr/lib/qiime/support_files/js'
+
+ if opts.output_dir:
+ if os.path.exists(opts.output_dir):
Added: trunk/packages/qiime/trunk/debian/patches/fix_t_test_bug
===================================================================
--- trunk/packages/qiime/trunk/debian/patches/fix_t_test_bug (rev 0)
+++ trunk/packages/qiime/trunk/debian/patches/fix_t_test_bug 2012-11-14 13:55:16 UTC (rev 12534)
@@ -0,0 +1,565 @@
+Version of the patch found here:
+https://github.com/jrrideout/qiime/commit/f9d6f4df9b227d23adf080ac012bd1331176d50c
+Extracted by cloning the repo and doing:
+git diff-tree -M -p f9d6f4d
+Then patch was manually merged and tested by running the test case.
+Note this patch introduces a dependency on the newer python-cogent.
+Patch can be dropped for 1.6 release.
+Author: Tim Booth <tbooth at ceh.ac.uk>
+
+---
+The information above should follow the Patch Tagging Guidelines, please
+checkout http://dep.debian.net/deps/dep3/ to learn about the format. Here
+are templates for supplementary fields that you might want to add:
+
+Origin: <vendor|upstream|other>, <url of original patch>
+Bug: <url in upstream bugtracker>
+Bug-Debian: http://bugs.debian.org/<bugnumber>
+Bug-Ubuntu: https://launchpad.net/bugs/<bugnumber>
+Forwarded: <no|not-needed|url proving that it has been forwarded>
+Reviewed-By: <name and email of someone who approved the patch>
+Last-Update: <YYYY-MM-DD>
+
+--- qiime-1.5.0+repack.orig/ChangeLog
++++ qiime-1.5.0+repack/ChangeLog
+@@ -1,3 +1,7 @@
++Patch backported from QIIME 1.6.0, dev branch
++==================================
++* compare_alpha_diversity.py now supports both parametric and nonparametric two sample t-tests (nonparametric is the default) with the new optional options -t/--test_type and -n/--num_permutations. Also fixed a bug that used the wrong degrees of freedom in the t-tests, yielding incorrect t statistics and p-values.
++
+ QIIME 1.5.0 (8 May 2012)
+ ==================================
+ * OTU tables are now stored on disk in the BIOM file format (see http://biom-format.org). The BIOM format webpage describes the motivation for the switch, but briefly it will support interoperability of related tools (e.g., QIIME/MG-RAST/mothur/VAMPS), and is a more efficient representation of data/metadata. The biom-format projects DenseTable and SparseTable objects are now used to represent OTU tables in memory. See the convert_biom.py script in the biom-format project for converting between 'classic' and BIOM formatted OTU tables.
+--- qiime-1.5.0+repack.orig/scripts/compare_alpha_diversity.py
++++ qiime-1.5.0+repack/scripts/compare_alpha_diversity.py
+@@ -4,7 +4,7 @@ from __future__ import division
+
+ __author__ = "William Van Treuren"
+ __copyright__ = "Copyright 2011, The QIIME project"
+-__credits__ = ["William Van Treuren, Greg Caparaso"]
++__credits__ = ["William Van Treuren", "Greg Caparaso", "Jai Ram Rideout"]
+ __license__ = "GPL"
+ __version__ = "1.5.0"
+ __maintainer__ = "William Van Treuren"
+@@ -13,80 +13,117 @@ __status__ = "Release"
+
+
+ from qiime.util import parse_command_line_parameters, make_option
+-from qiime.compare_alpha_diversity import compare_alpha_diversities,\
+- extract_rarefaction_scores_at_depth
++from qiime.compare_alpha_diversity import (compare_alpha_diversities,
++ extract_rarefaction_scores_at_depth,
++ test_types)
+
+ script_info = {}
+-script_info['brief_description'] = """This script compares alpha diversities based on a t_two_sample test"""
++script_info['brief_description'] = """This script compares alpha diversities based on a two sample t-test"""
+
+-script_info['script_description'] = """This script compares the alpha diversity of entries in a rarefaction file after they have been grouped based on some category found in the mapping file based on a t_two_sample test."""
++script_info['script_description'] = """
++This script compares the alpha diversity of entries in a rarefaction file after
++they have been grouped based on some category found in the mapping file based
++on a two sample t-test. The output file contains the
++(Category: (Subcategories): t, prob). By default the two sample t-test will be
++nonparametric (i.e. using Monte Carlo permutations to calculate the p-value),
++though the user has the option to make the test a parametric t-test.
++"""
+
+ script_info['script_usage'] = []
+-script_info['script_usage'].append(("""Explanation:\ Inputs: mapping file lines (lines of a mapping file which associates to each OTU/sample a number of characteristics, given as file path), rarefaction file lines (lines of a rarefaction file that has scores for each OTU/sample based on a certain rarefaction depth, given as a file path), depth (the depth score of the rarefaction file to use), category (the category to compare OTU/samples on), output file path (a path to the output file).""",\
+- """""",\
+- """%prog -i PD_whole_tree.txt -m map.txt -c Treatment -d 100 -o PD_d100.txt"""))
+-script_info['output_description']= \
+- """Script generates an output nested dictionary which has as a first
+- key:value pair the category passed, and a dictionary which gives the
+- t_two_sample score for every possible combination of the values
+- under that category in the mapping file, saved as a text file into
+- the directory specified by the output path."""
+
++script_info['script_usage'].append(("Comparing alpha diversities",
++"The following command takes the following input: a mapping file (which "
++"associaties each sample with a number of characteristics), alpha diversity "
++"metric (the results of collate_alpha for an alpha diverity metric, like "
++"PD_whole_tree), depth (the rarefaction depth to use for comparison), "
++"category (the category in the mapping file to determine which samples to "
++"compare to each other), and output file path (a path to the output file). A "
++"nonparametric two sample t-test is run to compare the alpha diversities "
++"using the default number of Monte Carlo permutations (999).",
++"%prog -i PD_whole_tree.txt -m mapping.txt -c Treatment -d 100 -o PD_d100.txt"))
++
++script_info['script_usage'].append(("Parametric t-test",
++"The following command runs a parametric two sample t-test using the "
++"t-distribution instead of Monte Carlo permutations.",
++"%prog -i PD_whole_tree.txt -m mapping.txt -c Treatment -d 100 -o "
++"PD_d100_parametric.txt -t parametric"))
++
++script_info['output_description']= """
++Script generates an output nested dictionary which has as a first key:value
++pair the category passed, and a dictionary which gives the t_two_sample score
++for every possible combination of the values under that category in the
++mapping file, saved as a text file into the directory specified by the output
++path.
++"""
+
+-script_info['required_options']=[\
++script_info['required_options']=[
+ make_option('-i',
+ '--alpha_diversity_filepath',
+ action='store',
+ type='existing_filepath',
+ dest='alpha_diversity_fp',
+- help='path to collated alpha diversity file (as generated by collate_alpha.py) [REQUIRED]'),\
++ help='path to collated alpha diversity file (as generated by '
++ 'collate_alpha.py) [REQUIRED]'),
+ make_option('-m',
+ '--mapping_filepath',
+ action='store',
+ type='existing_filepath',
+ dest='mapping_fp',
+- help='path to the mapping file [REQUIRED]'),\
++ help='path to the mapping file [REQUIRED]'),
+ make_option('-c',
+ '--category',
+ action='store',
+ type='string',
+ dest='category',
+- help='category for comparison [REQUIRED]'),\
++ help='category for comparison [REQUIRED]'),
+ make_option('-d',
+ '--depth',
+ action='store',
+ type='string',
+ dest='depth',
+- help='depth of rarefaction file to use [REQUIRED]'),\
++ help='depth of rarefaction file to use [REQUIRED]'),
+ make_option('-o',
+ '--output_filepath',
+ action='store',
+ type='new_filepath',
+ dest='output_fp',
+ help='output file path [REQUIRED]')]
++
++script_info['optional_options'] = [
++ make_option('-t', '--test_type', type='choice', choices=test_types,
++ help='the type of test to perform when calculating the p-values. Valid '
++ 'choices: ' + ', '.join(test_types) + '. If test_type is '
++ 'nonparametric, Monte Carlo permutations will be used to determine the '
++ 'p-value. If test_type is parametric, the num_permutations option will '
++ 'be ignored and the t-distribution will be used instead [default: '
++ '%default]', default='nonparametric'),
++ make_option('-n', '--num_permutations', type='int', default=999,
++ help='the number of permutations to perform when calculating the '
++ 'p-value. Must be greater than zero. Only applies if test_type is '
++ 'nonparametric [default: %default]')]
++
+ script_info['version'] = __version__
+
+ def main():
+ option_parser, opts, args = parse_command_line_parameters(**script_info)
++
+ rarefaction_lines = open(opts.alpha_diversity_fp, 'U')
+-
+ mapping_lines = open(opts.mapping_fp, 'U')
+ category = opts.category
+ depth = int(opts.depth)
+ output_path = opts.output_fp
+-
+-
+- result = compare_alpha_diversities(rarefaction_lines,\
+- mapping_lines,\
+- category,\
+- depth)
++
++ result = compare_alpha_diversities(rarefaction_lines, mapping_lines,
++ category, depth, opts.test_type,
++ opts.num_permutations)
+ outfile = open(output_path, 'w')
+ outfile.write(str(result))
+-
++ outfile.write('\n')
++
+ outfile.close()
+ rarefaction_lines.close()
+- mapping_lines.close()
+-
+-
++ mapping_lines.close()
++
++
+ if __name__ == "__main__":
+ main()
+--- qiime-1.5.0+repack.orig/qiime/compare_alpha_diversity.py
++++ qiime-1.5.0+repack/qiime/compare_alpha_diversity.py
+@@ -4,7 +4,7 @@ from __future__ import division
+
+ __author__ = "William Van Treuren"
+ __copyright__ = "Copyright 2011, The QIIME project"
+-__credits__ = ["William Van Treuren","Greg Caporaso"]
++__credits__ = ["William Van Treuren", "Greg Caporaso", "Jai Ram Rideout"]
+ __license__ = "GPL"
+ __version__ = "1.5.0"
+ __maintainer__ = "William Van Treuren"
+@@ -12,8 +12,11 @@ __email__ = "vantreur at colorado.edu"
+ __status__ = "Release"
+
+ from numpy import array, isnan
++from qiime.format import format_p_value_for_num_iters
+ from qiime.parse import parse_mapping_file_to_dict, parse_rarefaction
+-from cogent.maths.stats.test import t_two_sample
++from cogent.maths.stats.test import mc_t_two_sample, t_two_sample
++
++test_types = ['parametric', 'nonparametric']
+
+ def make_value_pairs_from_category(mapping_data, category):
+ """creates all pairs of unique category values from mapping data
+@@ -249,7 +252,8 @@ def convert_SampleIds_to_rarefaction_mtx
+
+
+ def compare_alpha_diversities(rarefaction_lines, mapping_lines,
+- category, depth):
++ category, depth, test_type='nonparametric',
++ num_permutations=999):
+ """compares alpha diversities
+
+ inputs:
+@@ -262,6 +266,12 @@ def compare_alpha_diversities(rarefactio
+ category - the category to be compared, is a string
+
+ depth - the depth of the rarefaction_file to use, is an integer
++
++ test_type - the type of t-test to perform, is a string. Must be either
++ 'parametric' or 'nonparametric'
++
++ num_permutations - the number of Monte Carlo permutations to use if
++ test_type is 'nonparametric', is an integer
+
+ outputs:
+ results - a nested dictionary which specifies the category as
+@@ -270,6 +280,9 @@ def compare_alpha_diversities(rarefactio
+ in the specified category
+
+ """
++ if test_type == 'nonparametric' and num_permutations < 1:
++ raise ValueError("Invalid number of permutations: %d. Must be greater "
++ "than zero." % num_permutations)
+
+ rarefaction_data = parse_rarefaction(rarefaction_lines)
+ mapping_data = parse_mapping_file_to_dict(mapping_lines)[0]
+@@ -290,15 +303,26 @@ def compare_alpha_diversities(rarefactio
+ results = {category:{}}
+
+ for pair in range(len(SampleId_pairs)):
++ # Must flatten the matrix because t_two_sample only operates on
++ # non-nested sequences (otherwise we'll get the wrong degrees of
++ # freedom).
+ i=(convert_SampleIds_to_rarefaction_mtx(SampleId_pairs[pair][0],
+- reduced_rarefaction_mtx, map_from_Id_to_col))
++ reduced_rarefaction_mtx,
++ map_from_Id_to_col)).flatten()
+
+ j=(convert_SampleIds_to_rarefaction_mtx(SampleId_pairs[pair][1],
+- reduced_rarefaction_mtx, map_from_Id_to_col))
+-
++ reduced_rarefaction_mtx,
++ map_from_Id_to_col)).flatten()
++
++ if test_type == 'parametric':
++ obs_t, p_val = t_two_sample(i,j)
++ elif test_type == 'nonparametric':
++ obs_t, _, _, p_val = mc_t_two_sample(i,j,
++ permutations=num_permutations)
++ p_val = format_p_value_for_num_iters(p_val, num_permutations)
++ else:
++ raise ValueError("Invalid test type '%s'." % test_type)
++
+ results[category][(str(value_pairs[pair][0]),
+- str(value_pairs[pair][1]))] =\
+- t_two_sample(i,j)
+-
++ str(value_pairs[pair][1]))] = obs_t, p_val
+ return results
+-
+--- qiime-1.5.0+repack.orig/tests/test_compare_alpha_diversities.py
++++ qiime-1.5.0+repack/tests/test_compare_alpha_diversities.py
+@@ -1,19 +1,16 @@
+ #!/usr/bin/env python
+ # File created on 19 May 2011
+
+-
+ from __future__ import division
+
+ __author__ = "William Van Treuren"
+ __copyright__ = "Copyright 2011, The QIIME project"
+-__credits__ = ["William Van Treuren, Greg Caporaso"]
++__credits__ = ["William Van Treuren", "Greg Caporaso", "Jai Ram Rideout"]
+ __license__ = "GPL"
+ __version__ = "1.5.0"
+ __maintainer__ = "William Van Treuren"
+ __email__ = "vantreur at colorado.edu"
+-__status__ = "Release"
+-
+-
++__status__ = "Development"
+
+ from cogent.util.unit_test import TestCase,main
+ from qiime.parse import parse_mapping_file_to_dict, parse_rarefaction
+@@ -32,13 +29,13 @@ class TopLevelTests(TestCase):
+ def setUp(self):
+ """define data for tests"""
+ self.rarefaction_file = \
+- ['\tsequences per sample\titeration\t123\t234\t345',
+- 'rare10.txt\t10\t0\t1.99181\t5.42877\t2.13996',
+- 'rare10.txt\t10\t1\t2.07163\t1.42877\t2.37055',
+- 'rare310.txt\t310\t0\t8.83115\t6.42877\t11.00725',
+- 'rare310.txt\t310\t1\t10.05242\t9.42877\t8.24474',
+- 'rare810.txt\t810\t0\t12.03067\tn/a\t11.58928',
+- 'rare910.txt\t910\t1\t12.9862\t2.42877\t11.58642']
++ ['\tsequences per sample\titeration\t123\t234\t345\t456',
++ 'rare10.txt\t10\t0\t1.99181\t5.42877\t2.13996\t0.002322',
++ 'rare10.txt\t10\t1\t2.07163\t1.42877\t2.37055\t0.01219',
++ 'rare310.txt\t310\t0\t8.83115\t6.42877\t11.00725\t0.18233',
++ 'rare310.txt\t310\t1\t10.05242\t9.42877\t8.24474\t0.99229',
++ 'rare810.txt\t810\t0\t12.03067\tn/a\t11.58928\t0.8993',
++ 'rare910.txt\t910\t1\t12.9862\t2.42877\t11.58642\t1.22563']
+
+ self.rarefaction_data = parse_rarefaction(self.rarefaction_file)
+
+@@ -48,7 +45,9 @@ class TopLevelTests(TestCase):
+ '#Comment Line',
+ '123\tAAAA\tBBBB\tHigh\t31\tM_ID_123',
+ '234\tCCCC\tDDDD\tLow\t67\tM_ID_234',
+- '345\tAAAA\tFFFF\tMed\t21\tM_ID_345']
++ '345\tAAAA\tFFFF\tMed\t21\tM_ID_345',
++ '456\tAAAA\tGGGG\tLow\t67\tM_ID_456'
++ ]
+
+ self.mapping_data = \
+ parse_mapping_file_to_dict(self.mapping_file)[0]
+@@ -59,32 +58,34 @@ class TopLevelTests(TestCase):
+ self.value_pairs_Treatment = \
+ [('CCCC', 'AAAA')]
+ self.cat_val_Dose = \
+- {'High': ['123'], 'Low': ['234'], 'Med': ['345']}
++ {'High': ['123'], 'Low': ['234', '456'], 'Med': ['345']}
+ self.cat_val_TTD = \
+- {'21': ['345'], '31': ['123'], '67': ['234']}
++ {'21': ['345'], '31': ['123'], '67': ['234', '456']}
+ self.cat_val_Treatment = \
+- {'AAAA': ['345', '123'], 'CCCC': ['234']}
++ {'AAAA': ['345', '123', '456'], 'CCCC': ['234']}
+ self.Id_pairs_Dose = \
+- [(['234'], ['345']), (['234'], ['123']), (['345'], ['123'])]
++ [(['234', '456'], ['345']), (['234', '456'], ['123']),
++ (['345'], ['123'])]
+ self.Id_pairs_TTD = \
+- [(['234'], ['345']), (['234'], ['123']), (['345'], ['123'])]
++ [(['234', '456'], ['345']), (['234', '456'], ['123']),
++ (['345'], ['123'])]
+
+ self.Id_pairs_Treatment = \
+- [(['234'], ['345', '123'])]
++ [(['234'], ['345', '123', '456'])]
+
+ self.rarefaction_cols_dict = \
+- {'123': 0, '234': 1, '345': 2}
++ {'123': 0, '234': 1, '345': 2, '456':3}
+
+ self.extracted_mtx_10 = \
+- array([[ 1.99181, 5.42877, 2.13996],
+- [ 2.07163, 1.42877, 2.37055]])
++ array([[ 1.99181, 5.42877, 2.13996, 0.002322],
++ [ 2.07163, 1.42877, 2.37055, 0.01219]])
+
+ self.extracted_mtx_310 = \
+- array([[ 8.83115, 6.42877, 11.00725],
+- [ 10.05242, 9.42877, 8.24474]])
++ array([[ 8.83115, 6.42877, 11.00725, 0.18233],
++ [ 10.05242, 9.42877, 8.24474, 0.99229]])
+
+ self.extracted_mtx_910 = \
+- array([[ 12.9862 , 2.42877, 11.58642]])
++ array([[ 12.9862 , 2.42877, 11.58642, 1.22563]])
+
+ self.sample_pair1 = \
+ (['234'], ['345', '123'])
+@@ -97,28 +98,26 @@ class TopLevelTests(TestCase):
+ array([[ 2.13996, 1.99181],
+ [ 2.37055, 2.07163]])
+
+- self.compared_alpha_diversities_TTD = \
+- {'TTD':{('21','31'):(1.8321466933860993,0.20839398129924847),
+- ('67', '21'): (0.58578495700890432, 0.61731739324369639),
+- ('67', '31'): (0.69838596448703294, 0.55721515283248324)}}
+-
++ self.compared_alpha_diversities_TTD = {'TTD': {('67', '21'):
++ (-0.27929839680103463, 0.79386220041241184), ('21', '31'):
++ (1.8321466933860993, 0.20839398129924847), ('67', '31'):
++ (-0.16318504125427058, 0.87828549279958279)}}
++
+ def test_make_value_pairs_from_category(self):
+ """check value pairs returns correct unique pairs for categories
+ """
+-
+-
++
+ self.assertEqual(
+- self.value_pairs_Dose,
+- make_value_pairs_from_category(self.mapping_data, 'Dose'))
++ make_value_pairs_from_category(self.mapping_data, 'Dose'),
++ self.value_pairs_Dose)
+
+ self.assertEqual(
+- self.value_pairs_TTD,
+- make_value_pairs_from_category(self.mapping_data, 'TTD'))
++ make_value_pairs_from_category(self.mapping_data, 'TTD'),
++ self.value_pairs_TTD)
+
+ self.assertEqual(
+- self.value_pairs_Treatment,
+- make_value_pairs_from_category(self.mapping_data,
+- 'Treatment'))
++ make_value_pairs_from_category(self.mapping_data, 'Treatment'),
++ self.value_pairs_Treatment)
+
+ self.assertRaises(
+ ValueError,
+@@ -128,59 +127,57 @@ class TopLevelTests(TestCase):
+ """check value pairs reference correct Id pairs"""
+
+ self.assertEqual(
+- self.cat_val_Dose,
+- make_category_values_Id_dict(self.mapping_data, 'Dose'))
++ make_category_values_Id_dict(self.mapping_data, 'Dose'),
++ self.cat_val_Dose)
+
+ self.assertEqual(
+- self.cat_val_TTD,
+- make_category_values_Id_dict(self.mapping_data, 'TTD'))
++ make_category_values_Id_dict(self.mapping_data, 'TTD'),
++ self.cat_val_TTD)
+
+ self.assertEqual(
+- self.cat_val_Treatment,
+- make_category_values_Id_dict(self.mapping_data,
+- 'Treatment'))
++ make_category_values_Id_dict(self.mapping_data, 'Treatment'),
++ self.cat_val_Treatment)
+
+ def test_map_category_value_pairs_to_Ids(self):
+ """check value pairs converted to correct Ids"""
+
+ self.assertEqual(
+- self.Id_pairs_Dose,
+ map_category_value_pairs_to_Ids(self.value_pairs_Dose,
+- self.cat_val_Dose))
++ self.cat_val_Dose),
++ self.Id_pairs_Dose)
+
+ self.assertEqual(
+- self.Id_pairs_TTD,
+ map_category_value_pairs_to_Ids(self.value_pairs_TTD,
+- self.cat_val_TTD))
++ self.cat_val_TTD),
++ self.Id_pairs_TTD)
+
+ self.assertEqual(
+- self.Id_pairs_Treatment,
+- map_category_value_pairs_to_Ids(
+- self.value_pairs_Treatment,
+- self.cat_val_Treatment))
++ map_category_value_pairs_to_Ids(self.value_pairs_Treatment,
++ self.cat_val_Treatment),
++ self.Id_pairs_Treatment)
+
+ def test_make_SampleIds_rarefaction_columns_dict(self):
+ """ """
+
+ self.assertEqual(
+- self.rarefaction_cols_dict,
+- make_SampleIds_rarefaction_columns_dict(self.rarefaction_data))
++ make_SampleIds_rarefaction_columns_dict(self.rarefaction_data),
++ self.rarefaction_cols_dict)
+
+ def test_extract_rarefaction_scores_at_depth(self):
+ """check correct errors raised for wrong depths and correct mtx
+ """
+
+- self.assertEqual(
+- self.extracted_mtx_10,
+- extract_rarefaction_scores_at_depth(10,self.rarefaction_data))
+-
+- self.assertEqual(
+- self.extracted_mtx_310,
+- extract_rarefaction_scores_at_depth(310,self.rarefaction_data))
+-
+- self.assertEqual(
+- self.extracted_mtx_910,
+- extract_rarefaction_scores_at_depth(910,self.rarefaction_data))
++ self.assertFloatEqual(
++ extract_rarefaction_scores_at_depth(10,self.rarefaction_data),
++ self.extracted_mtx_10)
++
++ self.assertFloatEqual(
++ extract_rarefaction_scores_at_depth(310,self.rarefaction_data),
++ self.extracted_mtx_310)
++
++ self.assertFloatEqual(
++ extract_rarefaction_scores_at_depth(910,self.rarefaction_data),
++ self.extracted_mtx_910)
+
+ self.assertRaises(
+ ValueError,
+@@ -215,13 +212,48 @@ class TopLevelTests(TestCase):
+ self.rarefaction_cols_dict),
+ self.rarefaction_mtx_for_sample_pair1_1)
+
+- def test_compare_alpha_diversity(self):
+- """test main function properly compares alpha diversities"""
++ def test_compare_alpha_diversities_parametric(self):
++ """test main function properly compares alpha divs (parametric)"""
+
+- self.assertEqual(
+- self.compared_alpha_diversities_TTD,
++ self.assertFloatEqual(
+ compare_alpha_diversities(self.rarefaction_file,
+- self.mapping_file, 'TTD', 10))
++ self.mapping_file, 'TTD', 10,
++ 'parametric'),
++ self.compared_alpha_diversities_TTD)
++
++ # Should ignore num_permutations if test_type is parametric.
++ self.assertFloatEqual(
++ compare_alpha_diversities(self.rarefaction_file,
++ self.mapping_file, 'TTD', 10,
++ 'parametric', 0),
++ self.compared_alpha_diversities_TTD)
++
++ def test_compare_alpha_diversities_nonparametric(self):
++ """test main function properly compares alpha divs (nonparametric)"""
++
++ obs = compare_alpha_diversities(self.rarefaction_file,
++ self.mapping_file, 'TTD', 10,
++ 'nonparametric')
++
++ # Since p-values are stochastic, we'll check that they are sane and
++ # that the t statistics are the same as we'd get for a parametric test.
++ for comp, (t, p_val) in obs['TTD'].items():
++ exp = self.compared_alpha_diversities_TTD['TTD'][comp]
++ self.assertFloatEqual(t, exp[0])
++ self.assertIsProb(float(p_val))
++
++ def test_compare_alpha_diversities_invalid_input(self):
++ """test main function bails on bad input"""
++
++ # Invalid test_type.
++ self.assertRaises(ValueError, compare_alpha_diversities,
++ self.rarefaction_file, self.mapping_file, 'TTD', 10,
++ 'foo')
++
++ # Invalid num_permutations.
++ self.assertRaises(ValueError, compare_alpha_diversities,
++ self.rarefaction_file, self.mapping_file, 'TTD', 10,
++ 'nonparametric', 0)
+
+
+ if __name__ == "__main__":
Added: trunk/packages/qiime/trunk/debian/patches/make_qiime_accept_new_rdp_classifier
===================================================================
--- trunk/packages/qiime/trunk/debian/patches/make_qiime_accept_new_rdp_classifier (rev 0)
+++ trunk/packages/qiime/trunk/debian/patches/make_qiime_accept_new_rdp_classifier 2012-11-14 13:55:16 UTC (rev 12534)
@@ -0,0 +1,72 @@
+This patch twists QIIME's arm to accept running a newer version
+of the RDP classifier by setting RDP_JAR_VERSION_OK, which is done
+by the QIIME wrapper.
+This is a nasty hack and hopefully the patch can be dropped for QIIME 1.6
+--- a/qiime/assign_taxonomy.py
++++ b/qiime/assign_taxonomy.py
+@@ -12,7 +12,7 @@
+ #import csv
+ import logging
+ import re
+-from os import system, remove, path, mkdir
++from os import system, remove, path, mkdir, getenv
+ from os.path import split, splitext
+ from glob import glob
+ from itertools import count
+@@ -45,9 +45,11 @@
+ "http://qiime.org/install/install.html#rdp-install")
+ elif "2.2" in rdp_jarpath:
+ return "rdp22"
++ elif getenv('RDP_JAR_VERSION_OK') is not None and float(getenv('RDP_JAR_VERSION_OK')) >= 2.2:
++ return "rdp22"
+ else:
+ raise ValueError(
+- "RDP classifier filename does not look like version 2.2. Only "
++ "RDP classifier filename does not look like version 2.2 or higher. Only "
+ "version 2.2 is supported by QIIME. RDP jar path is:"
+ " %s" % rdp_jarpath)
+
+@@ -306,7 +308,7 @@
+ """Assign taxon using RDP's naive Bayesian classifier
+ """
+ Name = "RdpTaxonAssigner"
+- Application = "RDP classfier, version 2.2"
++ Application = "RDP classfier, version 2.2+"
+ Citation = "Wang, Q, G. M. Garrity, J. M. Tiedje, and J. R. Cole. 2007. Naive Bayesian Classifier for Rapid Assignment of rRNA Sequences into the New Bacterial Taxonomy. Appl Environ Microbiol. 73(16):5261-7."
+ Taxonomy = "RDP"
+ _tracked_properties = ['Application','Citation','Taxonomy']
+--- a/scripts/assign_taxonomy.py
++++ b/scripts/assign_taxonomy.py
+@@ -14,7 +14,7 @@
+
+ from qiime.util import parse_command_line_parameters, get_options_lookup
+ from qiime.util import make_option, get_rdp_jarpath, load_qiime_config
+-from os import system, remove, path, mkdir
++from os import system, remove, path, mkdir, getenv
+ from os.path import split, splitext
+ from qiime.assign_taxonomy import (
+ BlastTaxonAssigner, RdpTaxonAssigner, RtaxTaxonAssigner, guess_rdp_version)
+@@ -205,6 +205,11 @@
+ params['training_data_properties_fp'] = opts.training_data_properties_fp
+ params['max_memory'] = "%sM" % opts.rdp_max_memory
+
++ #Allow RDP classifier to be >2.2, as hinted by an env variable
++ if getenv('RDP_JAR_VERSION_OK') is not None:
++ params['real_rdp_version'] = getenv('RDP_JAR_VERSION_OK')
++
++
+ elif assignment_method == 'rtax':
+ params['id_to_taxonomy_fp'] = opts.id_to_taxonomy_fp
+ params['reference_sequences_fp'] = opts.reference_seqs_fp
+--- a/scripts/make_3d_plots.py
++++ b/scripts/make_3d_plots.py
+@@ -397,7 +397,8 @@
+
+ qiime_dir=get_qiime_project_dir()
+
+- jar_path=os.path.join(qiime_dir,'qiime/support_files/jar/')
++ #jar_path=os.path.join(qiime_dir,'qiime/support_files/jar/')
++ jar_path='/usr/lib/qiime/support_files/jar/'
+
+ data_dir_path = get_random_directory_name(output_dir=dir_path,
+ return_absolute_path=False)
Modified: trunk/packages/qiime/trunk/debian/patches/series
===================================================================
--- trunk/packages/qiime/trunk/debian/patches/series 2012-11-14 10:09:03 UTC (rev 12533)
+++ trunk/packages/qiime/trunk/debian/patches/series 2012-11-14 13:55:16 UTC (rev 12534)
@@ -1,2 +1,5 @@
check_config_file_in_new_location.patch
fix_binary_helper_location.patch
+make_qiime_accept_new_rdp_classifier
+fix_path_for_support_files
+fix_t_test_bug
Modified: trunk/packages/qiime/trunk/debian/scripts/qiime
===================================================================
--- trunk/packages/qiime/trunk/debian/scripts/qiime 2012-11-14 10:09:03 UTC (rev 12533)
+++ trunk/packages/qiime/trunk/debian/scripts/qiime 2012-11-14 13:55:16 UTC (rev 12534)
@@ -27,9 +27,14 @@
fi
fi
+#Determine Qiime version:
+QIIME_VERSION=`/usr/lib/qiime/bin/print_qiime_config.py --version 2>/dev/null | awk '{print $NF}'`
+QIIME_VERSION=${QIIME_VERSION:-UNKNOWN}
+
echo """
-Setting up $QSNAME environment to run Qiime commands.
+Setting up $QSNAME environment to run QIIME commands.
+You have QIIME version $QIIME_VERSION.
Type 'help' for more info, 'exit' to return to regular shell.
"""
Modified: trunk/packages/qiime/trunk/debian/scripts/shell/qiime_environment
===================================================================
--- trunk/packages/qiime/trunk/debian/scripts/shell/qiime_environment 2012-11-14 10:09:03 UTC (rev 12533)
+++ trunk/packages/qiime/trunk/debian/scripts/shell/qiime_environment 2012-11-14 13:55:16 UTC (rev 12534)
@@ -14,6 +14,8 @@
# This was previously set in bio-linux-rdp-classifier but can now live here
# as the rdp-classifier package has been normalised.
export RDP_JAR_PATH=${RDP_JAR_PATH:-/usr/share/rdp-classifier/rdp_classifier.jar}
+#RDP_classifier is very bad at admitting what version it is, so ask DPKG:
+export RDP_JAR_VERSION_OK=`dpkg-query -f '${Version}\n' -W rdp-classifier | sed 's/[~+-].*//'`
# Mike Cox reports these are wanted for Ampliconnoise
export PYRO_LOOKUP_FILE=${PYRO_LOOKUP_FILE:-/usr/share/ampliconnoise/Data/LookUp_E123.dat}
More information about the debian-med-commit
mailing list