[med-svn] [Git][med-team/pybigwig][master] 8 commits: New upstream version 0.3.11
Diane Trout
gitlab at salsa.debian.org
Wed Jul 4 05:52:15 BST 2018
Diane Trout pushed to branch master at Debian Med / pybigwig
Commits:
7a3cc3ff by Diane Trout at 2018-07-03T14:48:55-07:00
New upstream version 0.3.11
- - - - -
5d5a8d3d by Diane Trout at 2018-07-03T14:48:55-07:00
Update upstream source from tag 'upstream/0.3.11'
Update to upstream version '0.3.11'
with Debian dir 3f2219c1b14f0d2ac3906d93eebdb7acba2acf82
- - - - -
f0e41a54 by Diane Trout at 2018-07-03T21:27:46-07:00
Remove disable-curl-check.patch. Upstream provided a good-enough solution
- - - - -
b9af9012 by Diane Trout at 2018-07-03T21:28:16-07:00
Word wrap Test-Command for easier reading
- - - - -
43b0b80f by Diane Trout at 2018-07-03T21:28:44-07:00
Remove ancient Python{,3}-Version tags
- - - - -
08460088 by Diane Trout at 2018-07-03T21:29:02-07:00
Update Standards-Version. No changes needed.
- - - - -
f86f47db by Diane Trout at 2018-07-03T21:42:47-07:00
Update debhelper to version 11
- - - - -
681380e8 by Diane Trout at 2018-07-03T21:43:40-07:00
Release to unstable
- - - - -
23 changed files:
- PKG-INFO
- README.md
- debian/changelog
- debian/compat
- debian/control
- − debian/patches/disable-curl-check.patch
- − debian/patches/series
- debian/tests/control
- libBigWig/README.md
- libBigWig/bigWig.h
- libBigWig/io.h → libBigWig/bigWigIO.h
- libBigWig/bwRead.c
- libBigWig/bwStats.c
- libBigWig/bwValues.c
- libBigWig/bwWrite.c
- libBigWig/io.c
- pyBigWig.c
- pyBigWig.egg-info/SOURCES.txt
- pyBigWig.h
- + pyBigWigTest/test.bigBed
- pyBigWigTest/test.py
- setup.cfg
- setup.py
Changes:
=====================================
PKG-INFO
=====================================
--- a/PKG-INFO
+++ b/PKG-INFO
@@ -1,12 +1,13 @@
Metadata-Version: 1.1
Name: pyBigWig
-Version: 0.3.2
+Version: 0.3.11
Summary: A package for accessing bigWig files using libBigWig
Home-page: https://github.com/dpryan79/pyBigWig
Author: Devon P. Ryan
Author-email: ryan at ie-freiburg.mpg.de
License: UNKNOWN
-Download-URL: https://github.com/dpryan79/pyBigWig/tarball/0.3.2
+Download-URL: https://github.com/dpryan79/pyBigWig/tarball/0.3.11
+Description-Content-Type: UNKNOWN
Description: UNKNOWN
Keywords: bioinformatics,bigWig,bigBed
Platform: UNKNOWN
=====================================
README.md
=====================================
--- a/README.md
+++ b/README.md
@@ -1,7 +1,7 @@
-[](https://badge.fury.io/py/pyBigWig) [](https://travis-ci.org/dpryan79/pyBigWig.svg?branch=master) [](http://bioconda.github.io) [](http://dx.doi.org/10.5281/zenodo.45238)
+[](https://badge.fury.io/py/pyBigWig) [](https://travis-ci.org/dpryan79/pyBigWig.svg?branch=master) [](http://bioconda.github.io) [](http://dx.doi.org/10.5281/zenodo.45238)
# pyBigWig
-A python extension, written in C, for quick access to and creation of bigWig files. This extension uses [libBigWig](https://github.com/dpryan79/libBigWig) for local and remote file access.
+A python extension, written in C, for quick access to bigBed files and access to and creation of bigWig files. This extension uses [libBigWig](https://github.com/dpryan79/libBigWig) for local and remote file access.
Table of Contents
=================
@@ -21,7 +21,9 @@ Table of Contents
* [Add a header to a bigWig file](#add-a-header-to-a-bigwig-file)
* [Adding entries to a bigWig file](#adding-entries-to-a-bigwig-file)
* [Close a bigWig or bigBed file](#close-a-bigwig-or-bigbed-file)
- * [Numpy](#Numpy)
+ * [Numpy](#numpy)
+ * [Remote file access](#remote-file-access)
+ * [Empty files](#empty-files)
* [A note on coordinates](#a-note-on-coordinates)
* [Galaxy](#galaxy)
@@ -296,6 +298,16 @@ Additionally, `values()` can directly output a numpy vector:
>>> type(bw.values('1', 0, 10, numpy=True))
<type 'numpy.ndarray'>
+# Remote file access
+
+If you do not have curl installed, pyBigWig will be installed without the ability to access remote files. You can determine if you will be able to access remote files with `pyBigWig.remote`. If that returns 1, then you can access remote files. If it returns 0 then you can't.
+
+# Empty files
+
+As of version 0.3.5, pyBigWig is able to read and write bigWig files lacking entries. Please note that such files are generally not compatible with other programs, since there's no definition of how a bigWig file with no entries should look. For such a file, the `intervals()` accessor will return `None`, the `stats()` function will return a list of `None` of the desired length, and `values()` will return `[]` (an empty list). This should generally allow programs utilizing pyBigWig to continue without issue.
+
+For those wishing to mimic the functionality of pyBigWig/libBigWig in this regard, please note that it looks at the number of bases covered (as reported in the file header) to check for "empty" files.
+
# A note on coordinates
Wiggle, bigWig, and bigBed files use 0-based half-open coordinates, which are also used by this extension. So to access the value for the first base on `chr1`, one would specify the starting position as `0` and the end position as `1`. Similarly, bases 100 to 115 would have a start of `99` and an end of `115`. This is simply for the sake of consistency with the underlying bigWig file and may change in the future.
=====================================
debian/changelog
=====================================
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,13 @@
+pybigwig (0.3.11-1) unstable; urgency=medium
+
+ * New upstream release.
+ * Word wrap Test-Command for easier reading
+ * Remove ancient Python{,3}-Version tags
+ * Update Standards-Version. No changes needed.
+ * Update debhelper to version 11
+
+ -- Diane Trout <diane at ghic.org> Tue, 03 Jul 2018 21:30:51 -0700
+
pybigwig (0.3.2-1) unstable; urgency=medium
* Initial release. (Closes: #846663)
=====================================
debian/compat
=====================================
--- a/debian/compat
+++ b/debian/compat
@@ -1 +1 @@
-9
+11
=====================================
debian/control
=====================================
--- a/debian/control
+++ b/debian/control
@@ -1,7 +1,7 @@
Source: pybigwig
Priority: optional
Maintainer: Diane Trout <diane at ghic.org>
-Build-Depends: debhelper (>= 9),
+Build-Depends: debhelper (>= 11),
dh-python,
libcurl4-gnutls-dev,
zlib1g-dev,
@@ -13,9 +13,7 @@ Build-Depends: debhelper (>= 9),
python3-all-dev,
python3-numpy,
python3-setuptools
-Standards-Version: 3.9.8
-X-Python-Version: >= 2.7
-X-Python3-Version: >= 3.3
+Standards-Version: 4.1.4
Section: science
Homepage: https://github.com/dpryan79/pyBigWig
=====================================
debian/patches/disable-curl-check.patch deleted
=====================================
--- a/debian/patches/disable-curl-check.patch
+++ /dev/null
@@ -1,26 +0,0 @@
-Author: Diane Trout <diane at ghic.org>
-Description: Don't call sys.exit when libcurl*-dev isn't available.
- This makes building the Debian source package a little easier.
-
---- a/setup.py
-+++ b/setup.py
-@@ -30,13 +30,13 @@
-
- try:
- foo, _ = subprocess.Popen(['curl-config', '--libs'], stdout=subprocess.PIPE).communicate()
--except:
-- sys.exit("Either libcurl isn't installed, it didn't come with curl-config, or curl-config isn't in your $PATH. This must be corrected before installing pyBigWig!\n")
-
--foo = foo.strip().split()
--for v in foo:
-- if(v[0:2] == "-L") :
-- additional_libs.append(v[2:])
-+ foo = foo.strip().split()
-+ for v in foo:
-+ if(v[0:2] == "-L") :
-+ additional_libs.append(v[2:])
-+except:
-+ print("Either libcurl isn't installed, it didn't come with curl-config, or curl-config isn't in your $PATH. This must be corrected before installing pyBigWig!\n")
-
- include_dirs = ['libBigWig', sysconfig.get_config_var("INCLUDEPY")]
- defines = []
=====================================
debian/patches/series deleted
=====================================
--- a/debian/patches/series
+++ /dev/null
@@ -1 +0,0 @@
-disable-curl-check.patch
=====================================
debian/tests/control
=====================================
--- a/debian/tests/control
+++ b/debian/tests/control
@@ -1,5 +1,15 @@
-Test-Command: set -e ; for py in $(pyversions -r 2>/dev/null) ; do cd pyBigWigTest; $py test.py ; cd .. ;done
+Test-Command: set -e
+ ; for py in $(pyversions -r 2>/dev/null)
+ ; do cd pyBigWigTest
+ ; $py test.py
+ ; cd ..
+ ; done
Depends: python-pybigwig
-Test-Command: set -e ; for py in $(py3versions -r 2>/dev/null) ; do cd pyBigWigTest; $py test.py ; cd .. ;done
+Test-Command: set -e
+ ; for py in $(py3versions -r 2>/dev/null)
+ ; do cd pyBigWigTest
+ ; $py test.py
+ ; cd ..
+ ; done
Depends: python3-pybigwig
=====================================
libBigWig/README.md
=====================================
--- a/libBigWig/README.md
+++ b/libBigWig/README.md
@@ -27,7 +27,7 @@ The only functions and structures that end users need to care about are in "bigW
//Open the local/remote file
fp = bwOpen(argv[1], NULL, "r");
if(!fp) {
- fprintf(stderr, "An error occured while opening %s\n", argv[1]);
+ fprintf(stderr, "An error occurred while opening %s\n", argv[1]);
return 1;
}
=====================================
libBigWig/bigWig.h
=====================================
--- a/libBigWig/bigWig.h
+++ b/libBigWig/bigWig.h
@@ -1,4 +1,4 @@
-#include "io.h"
+#include "bigWigIO.h"
#include "bwValues.h"
#include <inttypes.h>
#include <zlib.h>
@@ -53,7 +53,18 @@ extern "C" {
/*!
* The library version number
*/
-#define LIBBIGWIG_VERSION 0.3.0
+#define LIBBIGWIG_VERSION 0.4.2
+
+/*!
+ * If 1, then this library was compiled with remote file support.
+ */
+#ifdef NOCURL
+#define LIBBIGWIG_CURL 0
+typedef int CURLcode;
+typedef void CURL;
+#else
+#define LIBBIGWIG_CURL 1
+#endif
/*!
* The magic number of a bigWig file.
@@ -84,15 +95,16 @@ extern "C" {
* An enum that dictates the type of statistic to fetch for a given interval
*/
enum bwStatsType {
- doesNotExist = -1,
- mean = 0,
- average = 0,
- stdev = 1,
- dev = 1,
- max = 2,
- min = 3,
- cov = 4,
- coverage = 4
+ doesNotExist = -1, /*!< This does nothing */
+ mean = 0, /*!< The mean value */
+ average = 0, /*!< The mean value */
+ stdev = 1, /*!< The standard deviation of the values */
+ dev = 1, /*!< The standard deviation of the values */
+ max = 2, /*!< The maximum value */
+ min = 3, /*!< The minimum value */
+ cov = 4, /*!< The number of bases covered */
+ coverage = 4, /*!<The number of bases covered */
+ sum = 5 /*!< The sum of per-base values */
};
//Should hide this from end users
=====================================
libBigWig/io.h → libBigWig/bigWigIO.h
=====================================
--- a/libBigWig/io.h
+++ b/libBigWig/bigWigIO.h
@@ -1,5 +1,13 @@
+#ifndef NOCURL
#include <curl/curl.h>
-/*! \file io.h
+#else
+#include <stdio.h>
+typedef int CURLcode;
+typedef void CURL;
+#define CURLE_OK 0
+#define CURLE_FAILED_INIT 1
+#endif
+/*! \file bigWigIO.h
* These are (typically internal) IO functions, so there's generally no need for you to directly use them!
*/
@@ -23,7 +31,9 @@ enum bigWigFile_type_enum {
*/
typedef struct {
union {
+#ifndef NOCURL
CURL *curl; /**<The CURL * file pointer for remote files.*/
+#endif
FILE *fp; /**<The FILE * file pointer for local files.**/
} x; /**<A union holding curl and fp.*/
void *memBuf; /**<A void * pointing to memory of size bufSize.*/
@@ -32,6 +42,7 @@ typedef struct {
size_t bufSize; /**<The size of the buffer.*/
size_t bufLen; /**<The actual size of the buffer used.*/
enum bigWigFile_type_enum type; /**<The connection type*/
+ int isCompressed; /**<1 if the file is compressed, otherwise 0*/
char *fname; /**<Only needed for remote connections. The original URL/filename requested, since we need to make multiple connections.*/
} URL_t;
=====================================
libBigWig/bwRead.c
=====================================
--- a/libBigWig/bwRead.c
+++ b/libBigWig/bwRead.c
@@ -40,15 +40,19 @@ int bwInit(size_t defaultBufSize) {
GLOBAL_DEFAULTBUFFERSIZE = defaultBufSize;
//call curl_global_init()
+#ifndef NOCURL
CURLcode rv;
rv = curl_global_init(CURL_GLOBAL_ALL);
if(rv != CURLE_OK) return 1;
+#endif
return 0;
}
//This should be called before quiting, to release memory acquired by curl
void bwCleanup() {
+#ifndef NOCURL
curl_global_cleanup();
+#endif
}
static bwZoomHdr_t *bwReadZoomHdrs(bigWigFile_t *bw) {
@@ -153,6 +157,9 @@ static void bwHdrRead(bigWigFile_t *bw) {
if(bwRead((void*) &(bw->hdr->sumSquared), sizeof(uint64_t), 1, bw) != 1) goto error;
}
+ //In case of uncompressed remote files, let the IO functions know to request larger chunks
+ bw->URL->isCompressed = (bw->hdr->bufSize > 0)?1:0;
+
return;
error:
@@ -233,7 +240,7 @@ static uint64_t readChromBlock(bigWigFile_t *bw, chromList_t *cl, uint32_t keySi
static chromList_t *bwReadChromList(bigWigFile_t *bw) {
chromList_t *cl = NULL;
uint32_t magic, keySize, valueSize, itemsPerBlock;
- uint64_t i, rv, itemCount;
+ uint64_t rv, itemCount;
if(bw->isWrite) return NULL;
if(bwSetPos(bw, bw->hdr->ctOffset)) return NULL;
@@ -245,7 +252,7 @@ static chromList_t *bwReadChromList(bigWigFile_t *bw) {
if(bwRead((void*) &itemsPerBlock, sizeof(uint32_t), 1, bw) != 1) goto error;
if(bwRead((void*) &keySize, sizeof(uint32_t), 1, bw) != 1) goto error;
- if(bwRead((void*) &valueSize, sizeof(uint32_t), 1, bw) != 1) goto error; //Unused
+ if(bwRead((void*) &valueSize, sizeof(uint32_t), 1, bw) != 1) goto error;
if(bwRead((void*) &itemCount, sizeof(uint64_t), 1, bw) != 1) goto error;
cl->nKeys = itemCount;
@@ -254,16 +261,13 @@ static chromList_t *bwReadChromList(bigWigFile_t *bw) {
if(!cl->chrom) goto error;
if(!cl->len) goto error;
- if(bwRead((void*) &magic, sizeof(uint32_t), 1, bw) != 1) goto error; //padding
- if(bwRead((void*) &magic, sizeof(uint32_t), 1, bw) != 1) goto error; //padding
+ if(bwRead((void*) &magic, sizeof(uint32_t), 1, bw) != 1) goto error;
+ if(bwRead((void*) &magic, sizeof(uint32_t), 1, bw) != 1) goto error;
//Read in the blocks
- i = 0;
- while(i<itemCount) {
- rv = readChromBlock(bw, cl, keySize);
- if(rv == (uint64_t) -1) goto error;
- i += rv;
- }
+ rv = readChromBlock(bw, cl, keySize);
+ if(rv == (uint64_t) -1) goto error;
+ if(rv != itemCount) goto error;
return cl;
@@ -347,20 +351,33 @@ bigWigFile_t *bwOpen(char *fname, CURLcode (*callBack) (CURL*), const char *mode
if((!mode) || (strchr(mode, 'w') == NULL)) {
bwg->isWrite = 0;
bwg->URL = urlOpen(fname, *callBack, NULL);
- if(!bwg->URL) goto error;
+ if(!bwg->URL) {
+ fprintf(stderr, "[bwOpen] urlOpen is NULL!\n");
+ goto error;
+ }
//Attempt to read in the fixed header
bwHdrRead(bwg);
- if(!bwg->hdr) goto error;
+ if(!bwg->hdr) {
+ fprintf(stderr, "[bwOpen] bwg->hdr is NULL!\n");
+ goto error;
+ }
//Read in the chromosome list
bwg->cl = bwReadChromList(bwg);
- if(!bwg->cl) goto error;
+ if(!bwg->cl) {
+ fprintf(stderr, "[bwOpen] bwg->cl is NULL (%s)!\n", fname);
+ goto error;
+ }
//Read in the index
- bwg->idx = bwReadIndex(bwg, 0);
- if(!bwg->idx) goto error;
-
+ if(bwg->hdr->nBasesCovered) {
+ bwg->idx = bwReadIndex(bwg, 0);
+ if(!bwg->idx) {
+ fprintf(stderr, "[bwOpen] bwg->idx is NULL bwg->hdr->dataOffset 0x%"PRIx64"!\n", bwg->hdr->dataOffset);
+ goto error;
+ }
+ }
} else {
bwg->isWrite = 1;
bwg->URL = urlOpen(fname, NULL, "w+");
=====================================
libBigWig/bwStats.c
=====================================
--- a/libBigWig/bwStats.c
+++ b/libBigWig/bwStats.c
@@ -92,6 +92,7 @@ static struct vals_t *getVals(bigWigFile_t *fp, bwOverlapBlock_t *o, int i, uint
if(rv != Z_OK) goto error;
} else {
buf = compBuf;
+ sz = o->size[i];
}
p = buf;
@@ -123,12 +124,12 @@ static struct vals_t *getVals(bigWigFile_t *fp, bwOverlapBlock_t *o, int i, uint
free(v);
free(buf);
- free(compBuf);
+ if(compressed) free(compBuf);
return vals;
error:
if(buf) free(buf);
- if(compBuf) free(compBuf);
+ if(compBuf && compressed) free(compBuf);
if(v) free(v);
destroyVals_t(vals);
return NULL;
@@ -372,6 +373,52 @@ static double intCoverage(bwOverlappingIntervals_t* ints, uint32_t start, uint32
return o/(end-start);
}
+static double blockSum(bigWigFile_t *fp, bwOverlapBlock_t *blocks, uint32_t tid, uint32_t start, uint32_t end) {
+ uint32_t i, j, sizeUse;
+ double o = 0.0;
+ struct vals_t *v = NULL;
+
+ if(!blocks->n) return strtod("NaN", NULL);
+
+ //Iterate over the blocks
+ for(i=0; i<blocks->n; i++) {
+ v = getVals(fp, blocks, i, tid, start, end);
+ if(!v) goto error;
+ for(j=0; j<v->n; j++) {
+ //Multiply the block average by min(bases covered, block overlap with interval)
+ sizeUse = v->vals[j]->scalar;
+ if(sizeUse > v->vals[j]->nBases) sizeUse = v->vals[j]->nBases;
+ o+= (v->vals[j]->sum * sizeUse) / v->vals[j]->nBases;
+ }
+ destroyVals_t(v);
+ }
+
+ if(o == 0.0) return strtod("NaN", NULL);
+ return o;
+
+error:
+ destroyVals_t(v);
+ errno = ENOMEM;
+ return strtod("NaN", NULL);
+}
+
+static double intSum(bwOverlappingIntervals_t* ints, uint32_t start, uint32_t end) {
+ uint32_t i, start_use, end_use;
+ double o = 0.0;
+
+ if(!ints->l) return strtod("NaN", NULL);
+
+ for(i=0; i<ints->l; i++) {
+ start_use = ints->start[i];
+ end_use = ints->end[i];
+ if(start_use < start) start_use = start;
+ if(end_use > end) end_use = end;
+ o += (end_use - start_use) * ints->value[i];
+ }
+
+ return o;
+}
+
//Returns NULL on error, otherwise a double* that needs to be free()d
double *bwStatsFromZoom(bigWigFile_t *fp, int32_t level, uint32_t tid, uint32_t start, uint32_t end, uint32_t nBins, enum bwStatsType type) {
bwOverlapBlock_t *blocks = NULL;
@@ -382,6 +429,7 @@ double *bwStatsFromZoom(bigWigFile_t *fp, int32_t level, uint32_t tid, uint32_t
fp->hdr->zoomHdrs->idx[level] = bwReadIndex(fp, fp->hdr->zoomHdrs->indexOffset[level]);
if(!fp->hdr->zoomHdrs->idx[level]) return NULL;
}
+ errno = 0; //Sometimes libCurls sets and then doesn't unset errno on errors
output = malloc(sizeof(double)*nBins);
if(!output) return NULL;
@@ -412,6 +460,10 @@ double *bwStatsFromZoom(bigWigFile_t *fp, int32_t level, uint32_t tid, uint32_t
//cov
output[i] = blockCoverage(fp, blocks, tid, pos, end2)/(end2-pos);
break;
+ case 5:
+ //sum
+ output[i] = blockSum(fp, blocks, tid, pos, end2);
+ break;
default:
goto error;
break;
@@ -462,6 +514,9 @@ double *bwStatsFromFull(bigWigFile_t *fp, char *chrom, uint32_t start, uint32_t
case 4:
output[i] = intCoverage(ints, pos, end2);
break;
+ case 5:
+ output[i] = intSum(ints, pos, end2);
+ break;
}
bwDestroyOverlappingIntervals(ints);
pos = end2;
=====================================
libBigWig/bwValues.c
=====================================
--- a/libBigWig/bwValues.c
+++ b/libBigWig/bwValues.c
@@ -3,6 +3,7 @@
#include <stdlib.h>
#include <string.h>
#include <zlib.h>
+#include <errno.h>
static uint32_t roundup(uint32_t v) {
v--;
@@ -47,6 +48,9 @@ static bwRTree_t *readRTreeIdx(bigWigFile_t *fp, uint64_t offset) {
if(bwRead(&(node->blockSize), sizeof(uint32_t), 1, fp) != 1) goto error;
node->rootOffset = bwTell(fp);
+ //For remote files, libCurl sometimes sets errno to 115 and doesn't clear it
+ errno = 0;
+
return node;
error:
=====================================
libBigWig/bwWrite.c
=====================================
--- a/libBigWig/bwWrite.c
+++ b/libBigWig/bwWrite.c
@@ -90,19 +90,25 @@ static int writeAtPos(void *ptr, size_t sz, size_t nmemb, size_t pos, FILE *fp)
return 0;
}
-//Are nblocks and nperblock correct?
//We lose keySize bytes on error
static int writeChromList(FILE *fp, chromList_t *cl) {
uint16_t k;
uint32_t j, magic = CIRTREE_MAGIC;
- uint32_t nperblock = (cl->nKeys>0xFFFF)?-1:cl->nKeys; //Items per leaf/non-leaf
- uint32_t nblocks = (cl->nKeys>>16)+1, keySize = 0, valSize = 8; //does the valSize even matter? I ignore it...
- uint64_t i, written = 0;
+ uint32_t nperblock = (cl->nKeys > 0x7FFF) ? 0x7FFF : cl->nKeys; //Items per leaf/non-leaf, there are no unsigned ints in java :(
+ uint32_t nblocks, keySize = 0, valSize = 8; //In theory valSize could be optimized, in practice that'd be annoying
+ uint64_t i, nonLeafEnd, leafSize, nextLeaf;
uint8_t eight;
int64_t i64;
char *chrom;
size_t l;
+ if(cl->nKeys > 1073676289) {
+ fprintf(stderr, "[writeChromList] Error: Currently only 1,073,676,289 contigs are supported. If you really need more then please post a request on github.\n");
+ return 1;
+ }
+ nblocks = cl->nKeys/nperblock;
+ nblocks += ((cl->nKeys % nperblock) > 0)?1:0;
+
for(i64=0; i64<cl->nKeys; i64++) {
l = strlen(cl->chrom[i64]);
if(l>keySize) keySize = l;
@@ -122,32 +128,53 @@ static int writeChromList(FILE *fp, chromList_t *cl) {
if(fwrite(&i, sizeof(uint64_t), 1, fp) != 1) return 6;
//Do we need a non-leaf node?
- if(nblocks>1) {
+ if(nblocks > 1) {
eight = 0;
if(fwrite(&eight, sizeof(uint8_t), 1, fp) != 1) return 7;
if(fwrite(&eight, sizeof(uint8_t), 1, fp) != 1) return 8; //padding
- j = 0;
- for(i=0; i<nperblock; i++) { //Why yes, this is pointless
+ if(fwrite(&nblocks, sizeof(uint16_t), 1, fp) != 1) return 8;
+ nonLeafEnd = ftell(fp) + nperblock * (keySize + 8);
+ leafSize = nperblock * (keySize + 8) + 4;
+ for(i=0; i<nblocks; i++) { //Why yes, this is pointless
+ chrom = strncpy(chrom, cl->chrom[i * nperblock], keySize);
+ nextLeaf = nonLeafEnd + i * leafSize;
+ if(fwrite(chrom, keySize, 1, fp) != 1) return 9;
+ if(fwrite(&nextLeaf, sizeof(uint64_t), 1, fp) != 1) return 10;
+ }
+ for(i=0; i<keySize; i++) chrom[i] = '\0';
+ nextLeaf = 0;
+ for(i=nblocks; i<nperblock; i++) {
if(fwrite(chrom, keySize, 1, fp) != 1) return 9;
- if(fwrite(&j, sizeof(uint64_t), 1, fp) != 1) return 10;
+ if(fwrite(&nextLeaf, sizeof(uint64_t), 1, fp) != 1) return 10;
}
}
//Write the leaves
+ nextLeaf = 0;
for(i=0, j=0; i<nblocks; i++) {
eight = 1;
if(fwrite(&eight, sizeof(uint8_t), 1, fp) != 1) return 11;
eight = 0;
if(fwrite(&eight, sizeof(uint8_t), 1, fp) != 1) return 12;
- if(cl->nKeys - written < nperblock) nperblock = cl->nKeys - written;
- if(fwrite(&nperblock, sizeof(uint16_t), 1, fp) != 1) return 13;
+ if(cl->nKeys - j < nperblock) {
+ k = cl->nKeys - j;
+ if(fwrite(&k, sizeof(uint16_t), 1, fp) != 1) return 13;
+ } else {
+ if(fwrite(&nperblock, sizeof(uint16_t), 1, fp) != 1) return 13;
+ }
for(k=0; k<nperblock; k++) {
- if(j>=cl->nKeys) return 14;
- chrom = strncpy(chrom, cl->chrom[j], keySize);
- if(fwrite(chrom, keySize, 1, fp) != 1) return 15;
- if(fwrite(&j, sizeof(uint32_t), 1, fp) != 1) return 16;
- if(fwrite(&(cl->len[j++]), sizeof(uint32_t), 1, fp) != 1) return 17;
- written++;
+ if(j>=cl->nKeys) {
+ if(chrom[0]) {
+ for(l=0; l<keySize; l++) chrom[l] = '\0';
+ }
+ if(fwrite(chrom, keySize, 1, fp) != 1) return 15;
+ if(fwrite(&nextLeaf, sizeof(uint64_t), 1, fp) != 1) return 16;
+ } else {
+ chrom = strncpy(chrom, cl->chrom[j], keySize);
+ if(fwrite(chrom, keySize, 1, fp) != 1) return 15;
+ if(fwrite(&j, sizeof(uint32_t), 1, fp) != 1) return 16;
+ if(fwrite(&(cl->len[j++]), sizeof(uint32_t), 1, fp) != 1) return 17;
+ }
}
}
@@ -709,7 +736,7 @@ int writeIndex(bigWigFile_t *fp) {
bwLL *ll = fp->writeBuffer->firstIndexNode, *p;
bwRTreeNode_t *root = NULL;
- if(!fp->writeBuffer->nBlocks) return 1;
+ if(!fp->writeBuffer->nBlocks) return 0;
fp->idx = malloc(sizeof(bwRTree_t));
if(!fp->idx) return 2;
fp->idx->root = root;
@@ -780,7 +807,7 @@ int writeIndex(bigWigFile_t *fp) {
//This may or may not produce the requested number of zoom levels
int makeZoomLevels(bigWigFile_t *fp) {
uint32_t meanBinSize, i;
- uint32_t multiplier = 4, zoom = 10;
+ uint32_t multiplier = 4, zoom = 10, maxZoom = 0;
uint16_t nLevels = 0;
meanBinSize = ((double) fp->writeBuffer->runningWidthSum)/(fp->writeBuffer->nEntries);
@@ -801,7 +828,15 @@ int makeZoomLevels(bigWigFile_t *fp) {
if(!fp->hdr->zoomHdrs->indexOffset) return 4;
if(!fp->hdr->zoomHdrs->idx) return 5;
+ //There's no point in having a zoom level larger than the largest chromosome
+ //This will none the less allow at least one zoom level, which is generally needed for IGV et al.
+ for(i=0; i<fp->cl->nKeys; i++) {
+ if(fp->cl->len[i] > maxZoom) maxZoom = fp->cl->len[i];
+ }
+ if(zoom > maxZoom) zoom = maxZoom;
+
for(i=0; i<fp->hdr->nLevels; i++) {
+ if(zoom > maxZoom) break; //prevent absurdly large zoom levels
fp->hdr->zoomHdrs->level[i] = zoom;
nLevels++;
if(((uint32_t)-1)/multiplier < zoom) break;
@@ -887,6 +922,10 @@ uint32_t updateInterval(bigWigFile_t *fp, bwZoomBuffer_t *buffer, double *sum, d
uint32_t rv = 0, offset = 0;
if(!buffer) return 0;
if(buffer->l+32 >= buffer->m) return 0;
+
+ //Make sure that we don't overflow a uint32_t by adding some huge value to start
+ if(start + size < start) size = ((uint32_t) -1) - start;
+
if(buffer->l) {
offset = buffer->l/32;
} else {
@@ -1210,7 +1249,7 @@ int bwFinalize(bigWigFile_t *fp) {
if(writeIndex(fp)) return 4;
//Zoom level stuff here?
- if(fp->hdr->nLevels) {
+ if(fp->hdr->nLevels && fp->writeBuffer->nBlocks) {
offset = bwTell(fp);
if(makeZoomLevels(fp)) return 5;
if(constructZoomLevels(fp)) return 6;
=====================================
libBigWig/io.c
=====================================
--- a/libBigWig/io.c
+++ b/libBigWig/io.c
@@ -1,13 +1,17 @@
+#ifndef NOCURL
#include <curl/curl.h>
+#endif
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
-#include "io.h"
+#include "bigWigIO.h"
#include <inttypes.h>
+#include <errno.h>
size_t GLOBAL_DEFAULTBUFFERSIZE;
+#ifndef NOCURL
uint64_t getContentLength(URL_t *URL) {
double size;
if(curl_easy_getinfo(URL->x.curl, CURLINFO_CONTENT_LENGTH_DOWNLOAD, &size) != CURLE_OK) {
@@ -34,15 +38,17 @@ CURLcode urlFetchData(URL_t *URL, unsigned long bufSize) {
}
rv = curl_easy_perform(URL->x.curl);
+ errno = 0; //Sometimes curl_easy_perform leaves a random errno remnant
return rv;
}
//Read data into a buffer, ideally from a buffer already in memory
//The loop is likely no longer needed.
size_t url_fread(void *obuf, size_t obufSize, URL_t *URL) {
- size_t remaining = obufSize;
+ size_t remaining = obufSize, fetchSize;
void *p = obuf;
CURLcode rv;
+
while(remaining) {
if(!URL->bufLen) {
rv = urlFetchData(URL, URL->bufSize);
@@ -56,7 +62,12 @@ size_t url_fread(void *obuf, size_t obufSize, URL_t *URL) {
p += URL->bufLen - URL->bufPos;
remaining -= URL->bufLen - URL->bufPos;
if(remaining) {
- rv = urlFetchData(URL, (remaining<URL->bufSize)?remaining:URL->bufSize);
+ if(!URL->isCompressed) {
+ fetchSize = URL->bufSize;
+ } else {
+ fetchSize = (remaining<URL->bufSize)?remaining:URL->bufSize;
+ }
+ rv = urlFetchData(URL, fetchSize);
if(rv != CURLE_OK) {
fprintf(stderr, "[url_fread] urlFetchData (B) returned %s\n", curl_easy_strerror(rv));
return 0;
@@ -71,15 +82,20 @@ size_t url_fread(void *obuf, size_t obufSize, URL_t *URL) {
}
return obufSize;
}
+#endif
//Returns the number of bytes requested or a smaller number on error
//Note that in the case of remote files, the actual amount read may be less than the return value!
size_t urlRead(URL_t *URL, void *buf, size_t bufSize) {
+#ifndef NOCURL
if(URL->type==0) {
return fread(buf, bufSize, 1, URL->x.fp)*bufSize;
} else {
return url_fread(buf, bufSize, URL);
}
+#else
+ return fread(buf, bufSize, 1, URL->x.fp)*bufSize;
+#endif
}
size_t bwFillBuffer(void *inBuf, size_t l, size_t nmemb, void *pURL) {
@@ -102,15 +118,19 @@ size_t bwFillBuffer(void *inBuf, size_t l, size_t nmemb, void *pURL) {
//Seek to an arbitrary location, returning a CURLcode
//Note that a local file returns CURLE_OK on success or CURLE_FAILED_INIT on any error;
CURLcode urlSeek(URL_t *URL, size_t pos) {
+#ifndef NOCURL
char range[1024];
CURLcode rv;
if(URL->type == BWG_FILE) {
+#endif
if(fseek(URL->x.fp, pos, SEEK_SET) == 0) {
+ errno = 0;
return CURLE_OK;
} else {
return CURLE_FAILED_INIT; //This is arbitrary
}
+#ifndef NOCURL
} else {
//If the location is covered by the buffer then don't seek!
if(pos < URL->filePos || pos >= URL->filePos+URL->bufSize) {
@@ -128,29 +148,37 @@ CURLcode urlSeek(URL_t *URL, size_t pos) {
if(rv != CURLE_OK) {
fprintf(stderr, "[urlSeek] curl_easy_perform received an error!\n");
}
+ errno = 0; //Don't propogate remnant resolved libCurl errors
return rv;
} else {
URL->bufPos = pos-URL->filePos;
return CURLE_OK;
}
}
+#endif
}
URL_t *urlOpen(char *fname, CURLcode (*callBack)(CURL*), const char *mode) {
URL_t *URL = calloc(1, sizeof(URL_t));
if(!URL) return NULL;
char *url = NULL, *req = NULL;
+#ifndef NOCURL
CURLcode code;
char range[1024];
+#endif
URL->fname = fname;
if((!mode) || (strchr(mode, 'w') == 0)) {
//Set the protocol
+#ifndef NOCURL
if(strncmp(fname, "http://", 7) == 0) URL->type = BWG_HTTP;
else if(strncmp(fname, "https://", 8) == 0) URL->type = BWG_HTTPS;
else if(strncmp(fname, "ftp://", 6) == 0) URL->type = BWG_FTP;
else URL->type = BWG_FILE;
+#else
+ URL->type = BWG_FILE;
+#endif
//local file?
if(URL->type == BWG_FILE) {
@@ -161,6 +189,7 @@ URL_t *urlOpen(char *fname, CURLcode (*callBack)(CURL*), const char *mode) {
fprintf(stderr, "[urlOpen] Couldn't open %s for reading\n", fname);
return NULL;
}
+#ifndef NOCURL
} else {
//Remote file, set up the memory buffer and get CURL ready
URL->memBuf = malloc(GLOBAL_DEFAULTBUFFERSIZE);
@@ -175,6 +204,11 @@ URL_t *urlOpen(char *fname, CURLcode (*callBack)(CURL*), const char *mode) {
fprintf(stderr, "[urlOpen] curl_easy_init() failed!\n");
goto error;
}
+ //Negotiate a reasonable HTTP authentication method
+ if(curl_easy_setopt(URL->x.curl, CURLOPT_HTTPAUTH, CURLAUTH_ANY) != CURLE_OK) {
+ fprintf(stderr, "[urlOpen] Failed instructing curl to use any HTTP authentication it finds to be suitable!\n");
+ goto error;
+ }
//Follow redirects
if(curl_easy_setopt(URL->x.curl, CURLOPT_FOLLOWLOCATION, 1L) != CURLE_OK) {
fprintf(stderr, "[urlOpen] Failed instructing curl to follow redirects!\n");
@@ -200,6 +234,15 @@ URL_t *urlOpen(char *fname, CURLcode (*callBack)(CURL*), const char *mode) {
fprintf(stderr, "[urlOpen] Couldn't set CURLOPT_WRITEDATA!\n");
goto error;
}
+ //Ignore certificate errors with https, libcurl just isn't reliable enough with conda
+ if(curl_easy_setopt(URL->x.curl, CURLOPT_SSL_VERIFYPEER, 0) != CURLE_OK) {
+ fprintf(stderr, "[urlOpen] Couldn't set CURLOPT_SSL_VERIFYPEER to 0!\n");
+ goto error;
+ }
+ if(curl_easy_setopt(URL->x.curl, CURLOPT_SSL_VERIFYHOST, 0) != CURLE_OK) {
+ fprintf(stderr, "[urlOpen] Couldn't set CURLOPT_SSL_VERIFYHOST to 0!\n");
+ goto error;
+ }
if(callBack) {
code = callBack(URL->x.curl);
if(code != CURLE_OK) {
@@ -208,10 +251,12 @@ URL_t *urlOpen(char *fname, CURLcode (*callBack)(CURL*), const char *mode) {
}
}
code = curl_easy_perform(URL->x.curl);
+ errno = 0; //Sometimes curl_easy_perform leaves a random errno remnant
if(code != CURLE_OK) {
fprintf(stderr, "[urlOpen] curl_easy_perform received an error: %s\n", curl_easy_strerror(code));
goto error;
}
+#endif
}
} else {
URL->type = BWG_FILE;
@@ -226,6 +271,7 @@ URL_t *urlOpen(char *fname, CURLcode (*callBack)(CURL*), const char *mode) {
if(req) free(req);
return URL;
+#ifndef NOCURL
error:
if(url) free(url);
if(req) free(req);
@@ -233,15 +279,18 @@ error:
curl_easy_cleanup(URL->x.curl);
free(URL);
return NULL;
+#endif
}
//Performs the necessary free() operations and handles cleaning up curl
void urlClose(URL_t *URL) {
if(URL->type == BWG_FILE) {
fclose(URL->x.fp);
+#ifndef NOCURL
} else {
free(URL->memBuf);
curl_easy_cleanup(URL->x.curl);
+#endif
}
free(URL);
}
=====================================
pyBigWig.c
=====================================
--- a/pyBigWig.c
+++ b/pyBigWig.c
@@ -119,6 +119,7 @@ error:
return 0;
}
+//The calling function needs to free the result
char *getNumpyStr(PyArrayObject *obj, Py_ssize_t i) {
char *p , *o = NULL;
npy_intp stride, j;
@@ -137,7 +138,7 @@ char *getNumpyStr(PyArrayObject *obj, Py_ssize_t i) {
return o;
case NPY_UNICODE:
o = calloc(1, stride/4 + 1);
- for(j=0; j<stride/4; j++) o[j] = (char) ((uint32_t*)p)[4*j];
+ for(j=0; j<stride/4; j++) o[j] = (char) ((uint32_t*)p)[j];
return o;
default:
PyErr_SetString(PyExc_RuntimeError, "Received unknown data type!\n");
@@ -147,6 +148,25 @@ char *getNumpyStr(PyArrayObject *obj, Py_ssize_t i) {
}
#endif
+//Return 1 if there are any entries at all
+int hasEntries(bigWigFile_t *bw) {
+ if(bw->hdr->nBasesCovered > 0) return 1;
+ return 0;
+}
+
+PyObject* pyBwEnter(pyBigWigFile_t*self, PyObject *args) {
+ bigWigFile_t *bw = self->bw;
+
+ if(!bw) {
+ PyErr_SetString(PyExc_RuntimeError, "The bigWig file handle is not opened!");
+ return NULL;
+ }
+
+ Py_INCREF(self);
+
+ return (PyObject*) self;
+}
+
PyObject* pyBwOpen(PyObject *self, PyObject *pyFname) {
char *fname = NULL;
char *mode = "r";
@@ -161,13 +181,19 @@ PyObject* pyBwOpen(PyObject *self, PyObject *pyFname) {
} else {
bw = bbOpen(fname, NULL);
}
- if(!bw) goto error;
+ if(!bw) {
+ fprintf(stderr, "[pyBwOpen] bw is NULL!\n");
+ goto error;
+ }
if(!mode || !strchr(mode, 'w')) {
if(!bw->cl) goto error;
}
pybw = PyObject_New(pyBigWigFile_t, &bigWigFile);
- if(!pybw) goto error;
+ if(!pybw) {
+ fprintf(stderr, "[pyBwOpen] PyObject_New() returned NULL (out of memory?)!\n");
+ goto error;
+ }
pybw->bw = bw;
pybw->lastTid = -1;
pybw->lastType = -1;
@@ -199,6 +225,11 @@ static PyObject *pyBwGetHeader(pyBigWigFile_t *self, PyObject *args) {
bigWigFile_t *bw = self->bw;
PyObject *ret, *val;
+ if(!bw) {
+ PyErr_SetString(PyExc_RuntimeError, "The bigWig file handle is not opened!");
+ return NULL;
+ }
+
ret = PyDict_New();
val = PyLong_FromUnsignedLong(bw->hdr->version);
if(PyDict_SetItemString(ret, "version", val) == -1) goto error;
@@ -238,6 +269,11 @@ static PyObject *pyBwGetChroms(pyBigWigFile_t *self, PyObject *args) {
char *chrom = NULL;
uint32_t i;
+ if(!bw) {
+ PyErr_SetString(PyExc_RuntimeError, "The bigWig file handle is not opened!");
+ return NULL;
+ }
+
if(!(PyArg_ParseTuple(args, "|s", &chrom)) || !chrom) {
ret = PyDict_New();
for(i=0; i<bw->cl->nKeys; i++) {
@@ -276,6 +312,7 @@ enum bwStatsType char2enum(char *s) {
if(strcmp(s, "min") == 0) return min;
if(strcmp(s, "cov") == 0) return cov;
if(strcmp(s, "coverage") == 0) return cov;
+ if(strcmp(s, "sum") == 0) return sum;
return -1;
};
@@ -291,6 +328,11 @@ static PyObject *pyBwGetStats(pyBigWigFile_t *self, PyObject *args, PyObject *kw
int i, nBins = 1;
errno = 0; //In the off-chance that something elsewhere got an error and didn't clear it...
+ if(!bw) {
+ PyErr_SetString(PyExc_RuntimeError, "The bigWig file handle is not open!");
+ return NULL;
+ }
+
if(bw->type == 1) {
PyErr_SetString(PyExc_RuntimeError, "bigBed files have no statistics!");
return NULL;
@@ -322,6 +364,16 @@ static PyObject *pyBwGetStats(pyBigWigFile_t *self, PyObject *args, PyObject *kw
return NULL;
}
+ //Return a list of None if there are no entries at all
+ if(!hasEntries(bw)) {
+ ret = PyList_New(nBins);
+ for(i=0; i<nBins; i++) {
+ Py_INCREF(Py_None);
+ PyList_SetItem(ret, i, Py_None);
+ }
+ return ret;
+ }
+
//Get the actual statistics
if(exact == Py_True) {
val = bwStatsFromFull(bw, chrom, start, end, nBins, char2enum(type));
@@ -363,6 +415,11 @@ static PyObject *pyBwGetValues(pyBigWigFile_t *self, PyObject *args) {
PyObject *ret;
bwOverlappingIntervals_t *o;
+ if(!bw) {
+ PyErr_SetString(PyExc_RuntimeError, "The bigWig file handle is not open!");
+ return NULL;
+ }
+
if(bw->type == 1) {
PyErr_SetString(PyExc_RuntimeError, "bigBed files have no values! Use 'entries' instead.");
return NULL;
@@ -393,6 +450,18 @@ static PyObject *pyBwGetValues(pyBigWigFile_t *self, PyObject *args) {
return NULL;
}
+ if(!hasEntries(self->bw)) {
+#ifdef WITHNUMPY
+ if(outputNumpy == Py_True) {
+ return PyArray_SimpleNew(0, NULL, NPY_FLOAT);
+ } else {
+#endif
+ return PyList_New(0);
+#ifdef WITHNUMPY
+ }
+#endif
+ }
+
o = bwGetValues(self->bw, chrom, start, end, 1);
if(!o) {
PyErr_SetString(PyExc_RuntimeError, "An error occurred while fetching values!");
@@ -429,6 +498,11 @@ static PyObject *pyBwGetIntervals(pyBigWigFile_t *self, PyObject *args, PyObject
char *chrom;
PyObject *ret;
+ if(!bw) {
+ PyErr_SetString(PyExc_RuntimeError, "The bigWig file handle is not opened!");
+ return NULL;
+ }
+
if(bw->type == 1) {
PyErr_SetString(PyExc_RuntimeError, "bigBed files have no intervals! Use 'entries()' instead.");
return NULL;
@@ -453,6 +527,12 @@ static PyObject *pyBwGetIntervals(pyBigWigFile_t *self, PyObject *args, PyObject
return NULL;
}
+ //Check for empty files
+ if(!hasEntries(bw)) {
+ Py_INCREF(Py_None);
+ return Py_None;
+ }
+
//Get the intervals
intervals = bwGetOverlappingIntervals(bw, chrom, start, end);
if(!intervals) {
@@ -533,6 +613,11 @@ PyObject *pyBwAddHeader(pyBigWigFile_t *self, PyObject *args, PyObject *kwds) {
PyObject *InputTuple = NULL, *tmpObject, *tmpObject2;
Py_ssize_t i, pyLen;
+ if(!bw) {
+ PyErr_SetString(PyExc_RuntimeError, "The bigWig file handle is not open!");
+ return NULL;
+ }
+
if(!PyArg_ParseTupleAndKeywords(args, kwds, "O|k", kwd_list, &InputTuple, &zoomTmp)) {
PyErr_SetString(PyExc_RuntimeError, "Illegal arguments");
return NULL;
@@ -824,7 +909,7 @@ int canAppend(pyBigWigFile_t *self, int desiredType, PyObject *chroms, PyObject
uint32_t tid, uspan, ustep, ustart;
PyObject *tmp;
#ifdef WITHNUMPY
- void *foo;
+ char *chrom;
#endif
if(self->lastType == -1) return 0;
@@ -838,11 +923,13 @@ int canAppend(pyBigWigFile_t *self, int desiredType, PyObject *chroms, PyObject
#ifdef WITHNUMPY
if(PyArray_Check(chroms)) sz = PyArray_Size(chroms);
#endif
+
for(i=0; i<sz; i++) {
#ifdef WITHNUMPY
if(PyArray_Check(chroms)) {
- foo = PyArray_GETPTR1((PyArrayObject*)chroms, i);
- tid = bwGetTid(bw, PyString_AsString(PyArray_GETITEM((PyArrayObject*)chroms, foo)));
+ chrom = getNumpyStr((PyArrayObject*)chroms, i);
+ tid = bwGetTid(bw, chrom);
+ free(chrom);
} else {
#endif
tmp = PyList_GetItem(chroms, i);
@@ -852,6 +939,7 @@ int canAppend(pyBigWigFile_t *self, int desiredType, PyObject *chroms, PyObject
#endif
if(tid != (uint32_t) self->lastTid) return 0;
}
+
#ifdef WITHNUMPY
if(PyArray_Check(starts)) {
ustart = getNumpyU32((PyArrayObject*)starts, 0);
@@ -911,9 +999,6 @@ int PyAddIntervals(pyBigWigFile_t *self, PyObject *chroms, PyObject *starts, PyO
uint32_t n, *ustarts = NULL, *uends = NULL;
float *fvalues = NULL;
int rv;
-#ifdef WITHNUMPY
- void *foo;
-#endif
if(PyList_Check(starts)) sz = PyList_Size(starts);
#ifdef WITHNUMPY
@@ -933,8 +1018,7 @@ int PyAddIntervals(pyBigWigFile_t *self, PyObject *chroms, PyObject *starts, PyO
cchroms[i] = PyString_AsString(PyList_GetItem(chroms, i));
#ifdef WITHNUMPY
} else {
- foo = PyArray_GETPTR1((PyArrayObject*)chroms, i);
- cchroms[i] = PyString_AsString(PyArray_GETITEM((PyArrayObject*)chroms, foo));
+ cchroms[i] = getNumpyStr((PyArrayObject*)chroms, i);
#endif
}
if(PyList_Check(starts)) {
@@ -968,6 +1052,9 @@ int PyAddIntervals(pyBigWigFile_t *self, PyObject *chroms, PyObject *starts, PyO
self->lastTid = bwGetTid(bw, cchroms[n-1]);
self->lastStart = uends[n-1];
}
+ if(!PyList_Check(chroms)) {
+ for(i=0; i<n; i++) free(cchroms[i]);
+ }
free(cchroms);
free(ustarts);
free(uends);
@@ -1389,6 +1476,11 @@ PyObject *pyBwAddEntries(pyBigWigFile_t *self, PyObject *args, PyObject *kwds) {
PyObject *validate = Py_True;
int desiredType;
+ if(!self->bw) {
+ PyErr_SetString(PyExc_RuntimeError, "The bigWig file handle is not open!");
+ return NULL;
+ }
+
if(!PyArg_ParseTupleAndKeywords(args, kwds, "OO|OOOOO", kwd_list, &chroms, &starts, &ends, &values, &span, &step, &validate)) {
PyErr_SetString(PyExc_RuntimeError, "Illegal arguments");
return NULL;
@@ -1461,6 +1553,11 @@ static PyObject *pyBBGetEntries(pyBigWigFile_t *self, PyObject *args, PyObject *
int withString = 1;
bbOverlappingEntries_t *o;
+ if(!bw) {
+ PyErr_SetString(PyExc_RuntimeError, "The bigBed file handle is not open!");
+ return NULL;
+ }
+
if(bw->type == 0) {
PyErr_SetString(PyExc_RuntimeError, "bigWig files have no entries! Use 'intervals' or 'values' instead.");
return NULL;
@@ -1525,6 +1622,11 @@ static PyObject *pyBBGetSQL(pyBigWigFile_t *self, PyObject *args) {
size_t len = 0;
PyObject *o = NULL;
+ if(!bw) {
+ PyErr_SetString(PyExc_RuntimeError, "The bigBed file handle is not open!");
+ return NULL;
+ }
+
if(!str) {
Py_INCREF(Py_None);
return Py_None;
@@ -1554,6 +1656,12 @@ static PyObject *pyIsBigWig(pyBigWigFile_t *self, PyObject *args) {
static PyObject *pyIsBigBed(pyBigWigFile_t *self, PyObject *args) {
bigWigFile_t *bw = self->bw;
+
+ if(!bw) {
+ PyErr_SetString(PyExc_RuntimeError, "The bigBed file handle is not open!");
+ return NULL;
+ }
+
if(bw->type == 1) {
Py_INCREF(Py_True);
return Py_True;
@@ -1571,16 +1679,24 @@ static PyObject *pyIsBigBed(pyBigWigFile_t *self, PyObject *args) {
#if PY_MAJOR_VERSION >= 3
PyMODINIT_FUNC PyInit_pyBigWig(void) {
+#else
+PyMODINIT_FUNC initpyBigWig(void) {
+#endif
PyObject *res;
errno = 0; //just in case
+#if PY_MAJOR_VERSION >= 3
if(Py_AtExit(bwCleanup)) return NULL;
-
if(PyType_Ready(&bigWigFile) < 0) return NULL;
- if(Py_AtExit(bwCleanup)) return NULL;
if(bwInit(128000)) return NULL;
res = PyModule_Create(&pyBigWigmodule);
if(!res) return NULL;
+#else
+ if(Py_AtExit(bwCleanup)) return;
+ if(PyType_Ready(&bigWigFile) < 0) return;
+ if(bwInit(128000)) return;
+ res = Py_InitModule3("pyBigWig", bwMethods, "A module for handling bigWig files");
+#endif
Py_INCREF(&bigWigFile);
PyModule_AddObject(res, "pyBigWig", (PyObject *) &bigWigFile);
@@ -1592,25 +1708,14 @@ PyMODINIT_FUNC PyInit_pyBigWig(void) {
#else
PyModule_AddIntConstant(res, "numpy", 0);
#endif
-
- return res;
-}
+#ifdef NOCURL
+ PyModule_AddIntConstant(res, "remote", 0);
#else
-//Python2 initialization
-PyMODINIT_FUNC initpyBigWig(void) {
- PyObject *res;
- errno=0; //Sometimes libpython2.7.so is missing some links...
- if(Py_AtExit(bwCleanup)) return;
- if(PyType_Ready(&bigWigFile) < 0) return;
- if(bwInit(128000)) return; //This is temporary
- res = Py_InitModule3("pyBigWig", bwMethods, "A module for handling bigWig files");
+ PyModule_AddIntConstant(res, "remote", 1);
+#endif
+ PyModule_AddStringConstant(res, "__version__", pyBigWigVersion);
-#ifdef WITHNUMPY
- //Add the numpy constant
- import_array(); //Needed for numpy stuff to work
- PyModule_AddIntConstant(res, "numpy", 1);
-#else
- PyModule_AddIntConstant(res, "numpy", 0);
+#if PY_MAJOR_VERSION >= 3
+ return res;
#endif
}
-#endif
=====================================
pyBigWig.egg-info/SOURCES.txt
=====================================
--- a/pyBigWig.egg-info/SOURCES.txt
+++ b/pyBigWig.egg-info/SOURCES.txt
@@ -8,6 +8,7 @@ setup.py
libBigWig/LICENSE
libBigWig/README.md
libBigWig/bigWig.h
+libBigWig/bigWigIO.h
libBigWig/bwCommon.h
libBigWig/bwRead.c
libBigWig/bwStats.c
@@ -15,7 +16,7 @@ libBigWig/bwValues.c
libBigWig/bwValues.h
libBigWig/bwWrite.c
libBigWig/io.c
-libBigWig/io.h
pyBigWigTest/__init__.py
+pyBigWigTest/test.bigBed
pyBigWigTest/test.bw
pyBigWigTest/test.py
\ No newline at end of file
=====================================
pyBigWig.h
=====================================
--- a/pyBigWig.h
+++ b/pyBigWig.h
@@ -2,6 +2,8 @@
#include <structmember.h>
#include "bigWig.h"
+#define pyBigWigVersion "0.3.11"
+
typedef struct {
PyObject_HEAD
bigWigFile_t *bw;
@@ -13,6 +15,7 @@ typedef struct {
} pyBigWigFile_t;
static PyObject *pyBwOpen(PyObject *self, PyObject *pyFname);
+static PyObject *pyBwEnter(pyBigWigFile_t *self, PyObject *args);
static PyObject *pyBwClose(pyBigWigFile_t *pybw, PyObject *args);
static PyObject *pyBwGetChroms(pyBigWigFile_t *pybw, PyObject *args);
static PyObject *pyIsBigWig(pyBigWigFile_t *pybw, PyObject *args);
@@ -51,6 +54,10 @@ Arguments:\n\
\n\
>>> import pyBigWig\n\
>>> bw = pyBigWig.open(\"some_file.bw\")\n"},
+ {NULL, NULL, 0, NULL}
+};
+
+static PyMethodDef bwObjMethods[] = {
{"header", (PyCFunction)pyBwGetHeader, METH_VARARGS,
"Returns the header of a bigWig file. This contains information such as: \n\
* The version number of the file ('version').\n\
@@ -375,6 +382,8 @@ Keyword arguments:\n\
>>> bw.addEntries([\"1\", \"1\", \"1\"], [0, 100, 125], ends=[5, 120, 126], values=[0.0, 1.0, 200.0], validate=False)\n\
>>> bw.close()\n\
>>> os.remove(oname)"},
+ {"__enter__", (PyCFunction)pyBwEnter, METH_NOARGS, NULL},
+ {"__exit__", (PyCFunction)pyBwClose, METH_VARARGS, NULL},
{NULL, NULL, 0, NULL}
};
@@ -433,7 +442,7 @@ static PyTypeObject bigWigFile = {
0, /*tp_weaklistoffset*/
0, /*tp_iter*/
0, /*tp_iternext*/
- bwMethods, /*tp_methods*/
+ bwObjMethods, /*tp_methods*/
0, /*tp_members*/
0, /*tp_getset*/
0, /*tp_base*/
=====================================
pyBigWigTest/test.bigBed
=====================================
Binary files /dev/null and b/pyBigWigTest/test.bigBed differ
=====================================
pyBigWigTest/test.py
=====================================
--- a/pyBigWigTest/test.py
+++ b/pyBigWigTest/test.py
@@ -12,6 +12,10 @@ class TestRemote():
assert(bw is not None)
return bw
+ def doOpenWith(self):
+ with pyBigWig.open(self.fname) as bw:
+ assert(bw.chroms() == {'1': 195471971, '10': 130694993})
+
def doChroms(self, bw):
assert(bw.chroms() == {'1': 195471971, '10': 130694993})
assert(bw.chroms("1") == 195471971)
@@ -34,6 +38,9 @@ class TestRemote():
assert(bw.intervals("1", 0, 3) == ((0, 1, 0.10000000149011612), (1, 2, 0.20000000298023224), (2, 3, 0.30000001192092896)))
assert(bw.intervals("1") == ((0, 1, 0.10000000149011612), (1, 2, 0.20000000298023224), (2, 3, 0.30000001192092896), (100, 150, 1.399999976158142), (150, 151, 1.5)))
+ def doSum(self, bw):
+ assert(bw.stats("1", 100, 151, type="sum", nBins=2) == [35.0, 36.5])
+
def doWrite(self, bw):
ofile = tempfile.NamedTemporaryFile(delete=False)
oname = ofile.name
@@ -126,20 +133,74 @@ class TestRemote():
bw.close()
#check md5sum, this is the simplest method to check correctness
h = hashlib.md5(open(oname, "rb").read()).hexdigest()
- assert(h=="b1ca91d2ff42afdd2efa19a007c1ded4")
+ assert(h=="ef104f198c6ce8310acc149d0377fc16")
+ #Clean up
+ os.remove(oname)
+
+ def doWriteEmpty(self):
+ ofile = tempfile.NamedTemporaryFile(delete=False)
+ oname = ofile.name
+ ofile.close()
+ bw = pyBigWig.open(oname, "w")
+ bw.addHeader([("1", 1000000), ("2", 1500000)])
+ bw.close()
+
+ #check md5sum
+ h = hashlib.md5(open(oname, "rb").read()).hexdigest()
+ assert(h=="361c600e5badf0b45d819552a7822937")
+
+ #Ensure we can open and get reasonable results
+ bw = pyBigWig.open(oname)
+ assert(bw.chroms() == {'1': 1000000, '2': 1500000})
+ assert(bw.intervals("1") == None)
+ assert(bw.values("1", 0, 1000000) == [])
+ assert(bw.stats("1", 0, 1000000, nBins=2) == [None, None])
+ bw.close()
+
+ #Clean up
+ os.remove(oname)
+
+ def doWriteNumpy(self):
+ import numpy as np
+ ofile = tempfile.NamedTemporaryFile(delete=False)
+ oname = ofile.name
+ ofile.close()
+ bw = pyBigWig.open(oname, "w")
+ bw.addHeader([("chr1", 100), ("chr2", 150), ("chr3", 200), ("chr4", 250)])
+ chroms = np.array(["chr1"] * 2 + ["chr2"] * 2 + ["chr3"] * 2 + ["chr4"] * 2)
+ starts = np.array([0, 10, 40, 50, 60, 70, 80, 90], dtype=np.int64)
+ ends = np.array([5, 15, 45, 55, 65, 75, 85, 95], dtype=np.int64)
+ values0 = np.array(np.random.random_sample(8), dtype=np.float64)
+ bw.addEntries(chroms, starts, ends=ends, values=values0)
+ bw.close()
+
+ vals = [(x, y, z) for x, y, z in zip(starts, ends, values0)]
+ bw = pyBigWig.open(oname)
+ assert(bw.chroms() == {'chr1': 100, 'chr2': 150, 'chr3': 200, 'chr4': 250})
+ for idx1, chrom in enumerate(["chr1", "chr2", "chr3", "chr4"]):
+ for idx2, tup in enumerate(bw.intervals(chrom)):
+ assert(tup[0] == starts[2 * idx1 + idx2])
+ assert(tup[1] == ends[2 * idx1 + idx2])
+ assert(np.isclose(tup[2], values0[2 * idx1 + idx2]))
+ bw.close()
+
#Clean up
os.remove(oname)
def testAll(self):
bw = self.doOpen()
self.doChroms(bw)
- self.doHeader(bw)
- self.doStats(bw)
- self.doValues(bw)
- self.doIntervals(bw)
- self.doWrite(bw)
- if self.fname == "pyBigWigTest/test.bw":
+ if not self.fname.startswith("http"):
+ self.doHeader(bw)
+ self.doStats(bw)
+ self.doSum(bw)
+ self.doValues(bw)
+ self.doIntervals(bw)
+ self.doWrite(bw)
+ self.doOpenWith()
self.doWrite2()
+ self.doWriteEmpty()
+ self.doWriteNumpy()
bw.close()
class TestLocal():
@@ -150,7 +211,7 @@ class TestLocal():
class TestBigBed():
def testBigBed(self):
- fname = "http://www.encodeproject.org/files/ENCFF001JBR/@@download/ENCFF001JBR.bigBed"
+ fname = os.path.dirname(pyBigWig.__file__) + "/pyBigWigTest/test.bigBed"
bb = pyBigWig.open(fname)
assert(bb is not None)
assert(bb.isBigWig() == 0)
=====================================
setup.cfg
=====================================
--- a/setup.cfg
+++ b/setup.cfg
@@ -4,5 +4,4 @@ description-file = README.md
[egg_info]
tag_build =
tag_date = 0
-tag_svn_revision = 0
=====================================
setup.py
=====================================
--- a/setup.py
+++ b/setup.py
@@ -16,22 +16,28 @@ srcs = [x for x in
srcs.append("pyBigWig.c")
libs=["m", "z", "curl"]
-if sysconfig.get_config_vars('BLDLIBRARY') is not None:
- #Note the "-l" prefix!
- for e in sysconfig.get_config_vars('BLDLIBRARY')[0].split():
- if e[0:2] == "-l":
- libs.append(e[2:])
-elif(sys.version_info[0] >= 3 and sys.version_info[1] >= 3) :
- libs.append("python%i.%im" % (sys.version_info[0], sys.version_info[1]))
-else :
- libs.append("python%i.%i" % (sys.version_info[0], sys.version_info[1]))
+
+# do not link to python on mac, see https://github.com/deeptools/pyBigWig/issues/58
+if 'dynamic_lookup' not in (sysconfig.get_config_var('LDSHARED') or ''):
+ if sysconfig.get_config_vars('BLDLIBRARY') is not None:
+ #Note the "-l" prefix!
+ for e in sysconfig.get_config_vars('BLDLIBRARY')[0].split():
+ if e[0:2] == "-l":
+ libs.append(e[2:])
+ elif(sys.version_info[0] >= 3 and sys.version_info[1] >= 3) :
+ libs.append("python%i.%im" % (sys.version_info[0], sys.version_info[1]))
+ else :
+ libs.append("python%i.%i" % (sys.version_info[0], sys.version_info[1]))
additional_libs = [sysconfig.get_config_var("LIBDIR"), sysconfig.get_config_var("LIBPL")]
+defines = []
try:
foo, _ = subprocess.Popen(['curl-config', '--libs'], stdout=subprocess.PIPE).communicate()
except:
- sys.exit("Either libcurl isn't installed, it didn't come with curl-config, or curl-config isn't in your $PATH. This must be corrected before installing pyBigWig!\n")
+ foo = ''
+ defines.append(('NOCURL', None))
+ sys.stderr.write("Either libcurl isn't installed, it didn't come with curl-config, or curl-config isn't in your $PATH. pyBigWig will be installed without support for remote files.\n")
foo = foo.strip().split()
for v in foo:
@@ -39,7 +45,6 @@ for v in foo:
additional_libs.append(v[2:])
include_dirs = ['libBigWig', sysconfig.get_config_var("INCLUDEPY")]
-defines = []
if WITHNUMPY is True:
defines.extend([('WITHNUMPY', None), ('NPY_NO_DEPRECATED_API', 'NPY_1_7_API_VERSION')])
extra_info = get_info('npymath')
@@ -56,12 +61,12 @@ module1 = Extension('pyBigWig',
include_dirs = include_dirs)
setup(name = 'pyBigWig',
- version = '0.3.2',
+ version = '0.3.11',
description = 'A package for accessing bigWig files using libBigWig',
author = "Devon P. Ryan",
author_email = "ryan at ie-freiburg.mpg.de",
url = "https://github.com/dpryan79/pyBigWig",
- download_url = "https://github.com/dpryan79/pyBigWig/tarball/0.3.2",
+ download_url = "https://github.com/dpryan79/pyBigWig/tarball/0.3.11",
keywords = ["bioinformatics", "bigWig", "bigBed"],
classifier = ["Development Status :: 5 - Production/Stable",
"Intended Audience :: Developers",
@@ -74,6 +79,7 @@ setup(name = 'pyBigWig',
"Programming Language :: Python :: 3.3",
"Programming Language :: Python :: 3.4",
"Programming Language :: Python :: 3.5",
+ "Programming Language :: Python :: 3.6",
"Programming Language :: Python :: Implementation :: CPython",
"Operating System :: POSIX",
"Operating System :: Unix",
View it on GitLab: https://salsa.debian.org/med-team/pybigwig/compare/8529a00b0ad3648eba71265bb721b1f46a6e50a9...681380e8787e31eac99595322c89261a699d3c22
--
View it on GitLab: https://salsa.debian.org/med-team/pybigwig/compare/8529a00b0ad3648eba71265bb721b1f46a6e50a9...681380e8787e31eac99595322c89261a699d3c22
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/debian-med-commit/attachments/20180704/6fdfb800/attachment-0001.html>
More information about the debian-med-commit
mailing list