[med-svn] [Git][med-team/deepnano][master] 15 commits: d/t/run-test.sh: don't meddle with normal user's home.
Étienne Mollier (@emollier)
gitlab at salsa.debian.org
Sun Dec 18 20:30:19 GMT 2022
Étienne Mollier pushed to branch master at Debian Med / deepnano
Commits:
658961f9 by Étienne Mollier at 2022-12-18T18:11:26+01:00
d/t/run-test.sh: don't meddle with normal user's home.
- - - - -
5955f356 by Étienne Mollier at 2022-12-18T18:45:45+01:00
fix-TypeError.patch: add; fix autopkgtest failure.
- - - - -
9ab07519 by Étienne Mollier at 2022-12-18T19:34:38+01:00
py3-compat.patch: don't undo align_2d path patching.
- - - - -
691ca783 by Étienne Mollier at 2022-12-18T19:51:32+01:00
d/t/run-test.sh: cap OpenMP threads to 16.
This is a maneuver to avoid #921566 hopefully.
- - - - -
a60bcba7 by Étienne Mollier at 2022-12-18T19:55:12+01:00
d/t/run-test.sh: timeout commands early in case of hangup.
Closes: #921566
- - - - -
8f80bfd6 by Étienne Mollier at 2022-12-18T21:02:39+01:00
basecall_no_metrichor.patch: add; fix python2-ism TypeError.
- - - - -
f1e0cf0f by Étienne Mollier at 2022-12-18T21:16:44+01:00
initialise changelog.
- - - - -
75939d1a by Étienne Mollier at 2022-12-18T21:18:27+01:00
routine-update: Standards-Version: 4.6.2
- - - - -
f51225d7 by Étienne Mollier at 2022-12-18T21:18:27+01:00
routine-update: debhelper-compat 13
- - - - -
92a4f59b by Étienne Mollier at 2022-12-18T21:18:30+01:00
routine-update: Add salsa-ci file
- - - - -
95168a9c by Étienne Mollier at 2022-12-18T21:18:31+01:00
routine-update: Rules-Requires-Root: no
- - - - -
a75bc1f5 by Étienne Mollier at 2022-12-18T21:18:38+01:00
Set upstream metadata fields: Repository.
Changes-By: lintian-brush
- - - - -
f66b0d57 by Étienne Mollier at 2022-12-18T21:18:42+01:00
Apply multi-arch hints.
+ deepnano-data: Add Multi-Arch: foreign.
Changes-By: apply-multiarch-hints
- - - - -
68f4d8dc by Étienne Mollier at 2022-12-18T21:19:36+01:00
routine-update: watch file standard 4
- - - - -
4da728bf by Étienne Mollier at 2022-12-18T21:25:41+01:00
routine-update: Ready to upload to unstable
- - - - -
11 changed files:
- debian/changelog
- − debian/compat
- debian/control
- + debian/patches/basecall_no_metrichor.patch
- + debian/patches/fix-TypeError.patch
- debian/patches/py3-compat.patch
- debian/patches/series
- + debian/salsa-ci.yml
- debian/tests/run-test.sh
- debian/upstream/metadata
- debian/watch
Changes:
=====================================
debian/changelog
=====================================
@@ -1,3 +1,23 @@
+deepnano (0.0+git20170813.e8a621e-4) unstable; urgency=medium
+
+ * Team upload.
+ * d/t/run-test.sh: don't meddle with normal user's home.
+ * fix-TypeError.patch: add; fix autopkgtest failure.
+ * py3-compat.patch: don't undo align_2d path patching.
+ * d/t/run-test.sh: cap OpenMP threads to 16.
+ * d/t/run-test.sh: timeout commands early in case of hangup.
+ Closes: #921566
+ * basecall_no_metrichor.patch: add; open align_2d in text mode, this
+ avoids type error due to writing text in a binary opened file.
+ * Standards-Version: 4.6.2 (routine-update)
+ * debhelper-compat 13 (routine-update)
+ * Add salsa-ci file (routine-update)
+ * Rules-Requires-Root: no (routine-update)
+ * Set upstream metadata fields: Repository.
+ * watch file standard 4 (routine-update)
+
+ -- Étienne Mollier <emollier at debian.org> Sun, 18 Dec 2022 21:19:36 +0100
+
deepnano (0.0+git20170813.e8a621e-3.1) unstable; urgency=medium
* Non-maintainer upload.
=====================================
debian/compat deleted
=====================================
@@ -1 +0,0 @@
-12
=====================================
debian/control
=====================================
@@ -4,15 +4,16 @@ Uploaders: Çağrı ULAŞ <cagriulas at gmail.com>,
Andreas Tille <tille at debian.org>
Section: science
Priority: optional
-Build-Depends: debhelper (>= 12~),
+Build-Depends: debhelper-compat (= 13),
python3-all,
dh-python,
python3-markdown,
python3-theano
-Standards-Version: 4.3.0
+Standards-Version: 4.6.2
Vcs-Browser: https://salsa.debian.org/med-team/deepnano
Vcs-Git: https://salsa.debian.org/med-team/deepnano.git
Homepage: https://bitbucket.org/vboza/deepnano
+Rules-Requires-Root: no
Package: deepnano
Architecture: any
@@ -36,6 +37,7 @@ Package: deepnano-data
Architecture: all
Depends: ${misc:Depends}
Suggests: deepnano
+Multi-Arch: foreign
Description: alternative basecaller for MinION reads of genomic sequences (data)
DeepNano is alternative basecaller for Oxford Nanopore MinION reads
based on deep recurrent neural networks.
=====================================
debian/patches/basecall_no_metrichor.patch
=====================================
@@ -0,0 +1,36 @@
+Description:
+ In python3, there is a hard distinction between files opened in text mode,
+ and files opened in binary mode. By default, subprocess.Popen will open
+ files in binary mode, but subsequent calls to print writing to the opened
+ pipelines will fail with:
+ .
+ Traceback (most recent call last):
+ File "/usr/share/deepnano/basecall_no_metrichor.py", line 245, in <module>
+ basecall(read, fo)
+ File "/usr/share/deepnano/basecall_no_metrichor.py", line 162, in basecall
+ print(len(o1)+len(o2), file=f2d)
+ TypeError: a bytes-like object is required, not 'str'
+ .
+ Setting the text mode in subprocess.Popen will circumvent that.
+Author: Étienne Mollier <emollier at debian.org>
+Forwarded: no
+Last-Update: 2022-12-18
+---
+This patch header follows DEP-3: http://dep.debian.net/deps/dep3/
+--- deepnano.orig/basecall_no_metrichor.py
++++ deepnano/basecall_no_metrichor.py
+@@ -157,9 +157,12 @@
+ if do_2d and "comp_events2" in data and\
+ len(data["comp_events2"]) <= args.max_2d_length and\
+ len(data["temp_events2"]) <= args.max_2d_length:
+- p = subprocess.Popen("/usr/lib/deepnano/align_2d", stdin=subprocess.PIPE, stdout=subprocess.PIPE)
++ p = subprocess.Popen("/usr/lib/deepnano/align_2d",
++ stdin=subprocess.PIPE,
++ stdout=subprocess.PIPE,
++ text=True)
+ f2d = p.stdin
+- print(len(o1)+len(o2), file=f2d)
++ print(len(o1) + len(o2), file=f2d)
+ for a, b in zip(o1, o2):
+ print(" ".join(map(str, a)), file=f2d)
+ print(" ".join(map(str, b)), file=f2d)
=====================================
debian/patches/fix-TypeError.patch
=====================================
@@ -0,0 +1,28 @@
+Description: use integer division to fix subsequent TypeError.
+ If this is not addressed, the autopkgtest fails early with the following
+ error:
+ .
+ #1 - deepnano_basecall
+ loading template net
+ Traceback (most recent call last):
+ File "/usr/share/deepnano/basecall.py", line 105, in <module>
+ temp_net = RnnPredictor(args.template_net)
+ File "/usr/share/deepnano/rnn_fin.py", line 62, in __init__
+ for i in range(n_layers):
+ TypeError: 'float' object cannot be interpreted as an integer
+Author: Étienne Mollier <emollier at debian.org>
+Forwarded: no
+Last-Update: 2022-12-18
+---
+This patch header follows DEP-3: http://dep.debian.net/deps/dep3/
+--- deepnano.orig/rnn_fin.py
++++ deepnano/rnn_fin.py
+@@ -52,7 +52,7 @@
+ def __init__(self, filename):
+ package = np.load(filename)
+ assert(len(package.files) % 20 == 4)
+- n_layers = len(package.files) / 20
++ n_layers = len(package.files) // 20
+
+ self.input = T.fmatrix()
+ last_output = self.input
=====================================
debian/patches/py3-compat.patch
=====================================
@@ -3,8 +3,6 @@ Author: Thomas Goirand <zigo at debian.org>
Forwarded: no
Last-Update: 2019-11-03
-Index: deepnano/basecall.py
-===================================================================
--- deepnano.orig/basecall.py
+++ deepnano/basecall.py
@@ -1,4 +1,5 @@
@@ -13,7 +11,7 @@ Index: deepnano/basecall.py
import argparse
from rnn_fin import RnnPredictor
import h5py
-@@ -66,7 +67,7 @@ def load_read_data(read_file):
+@@ -66,7 +67,7 @@
ret["2d_events"].append(ev)
ret["2d_events"] = np.array(ret["2d_events"], dtype=np.float32)
except Exception as e:
@@ -22,7 +20,7 @@ Index: deepnano/basecall.py
pass
h5.close()
-@@ -100,17 +101,17 @@ assert do_template or do_complement or d
+@@ -100,17 +101,17 @@
assert len(args.reads) != 0 or len(args.directory) != 0, "Nothing to basecall"
if do_template:
@@ -46,7 +44,7 @@ Index: deepnano/basecall.py
chars = "ACGT"
mapping = {"A": 0, "C": 1, "G": 2, "T": 3, "N": 4}
-@@ -128,24 +129,24 @@ for i, read in enumerate(files):
+@@ -128,24 +129,24 @@
try:
data = load_read_data(read)
except Exception as e:
@@ -80,7 +78,7 @@ Index: deepnano/basecall.py
except:
pass
-@@ -166,18 +167,18 @@ for i, read in enumerate(files):
+@@ -166,18 +167,18 @@
if args.timing:
try:
@@ -106,8 +104,6 @@ Index: deepnano/basecall.py
except:
# Don't let timing throw us out
pass
-Index: deepnano/basecall_no_metrichor.py
-===================================================================
--- deepnano.orig/basecall_no_metrichor.py
+++ deepnano/basecall_no_metrichor.py
@@ -1,4 +1,5 @@
@@ -116,7 +112,7 @@ Index: deepnano/basecall_no_metrichor.py
import argparse
from rnn_fin import RnnPredictor
import h5py
-@@ -82,7 +83,7 @@ def load_read_data(read_file):
+@@ -82,7 +83,7 @@
h5 = h5py.File(read_file, "r")
ret = {}
@@ -125,7 +121,7 @@ Index: deepnano/basecall_no_metrichor.py
base_events = h5["Analyses/EventDetection_000/Reads"][read_key]["Events"]
temp_comp_loc = template_complement_loc(base_events)
sampling_rate = h5["UniqueGlobalKey/channel_id"].attrs["sampling_rate"]
-@@ -137,8 +138,8 @@ def basecall(read_file_name, fo):
+@@ -137,8 +138,8 @@
try:
data = load_read_data(read_file_name)
except Exception as e:
@@ -136,12 +132,9 @@ Index: deepnano/basecall_no_metrichor.py
return
if do_template or do_2d:
-@@ -156,22 +157,22 @@ def basecall(read_file_name, fo):
- if do_2d and "comp_events2" in data and\
- len(data["comp_events2"]) <= args.max_2d_length and\
+@@ -158,20 +159,20 @@
len(data["temp_events2"]) <= args.max_2d_length:
-- p = subprocess.Popen("/usr/lib/deepnano/align_2d", stdin=subprocess.PIPE, stdout=subprocess.PIPE)
-+ p = subprocess.Popen("./align_2d", stdin=subprocess.PIPE, stdout=subprocess.PIPE)
+ p = subprocess.Popen("/usr/lib/deepnano/align_2d", stdin=subprocess.PIPE, stdout=subprocess.PIPE)
f2d = p.stdin
- print >>f2d, len(o1)+len(o2)
+ print(len(o1)+len(o2), file=f2d)
@@ -168,7 +161,7 @@ Index: deepnano/basecall_no_metrichor.py
events_2d = []
for l in lines[1:]:
temp_ind, comp_ind = map(int, l.strip().split())
-@@ -218,17 +219,17 @@ assert do_template or do_complement or d
+@@ -218,17 +219,17 @@
assert len(args.reads) != 0 or len(args.directory) != 0 or len(args.watch) != 0, "Nothing to basecall"
if do_template or do_2d:
@@ -192,7 +185,7 @@ Index: deepnano/basecall_no_metrichor.py
chars = "ACGT"
mapping = {"A": 0, "C": 1, "G": 2, "T": 3, "N": 4}
-@@ -250,14 +251,14 @@ if len(args.watch) != 0:
+@@ -250,14 +251,14 @@
from watchdog.observers import Observer
from watchdog.events import PatternMatchingEventHandler
except:
@@ -209,8 +202,6 @@ Index: deepnano/basecall_no_metrichor.py
file_name = str(os.path.basename(event.src_path))
fasta_file_name = os.path.splitext(event.src_path)[0] + '.fasta'
with open(fasta_file_name, "w") as fo:
-Index: deepnano/basecall_no_metrichor_devel.py
-===================================================================
--- deepnano.orig/basecall_no_metrichor_devel.py
+++ deepnano/basecall_no_metrichor_devel.py
@@ -1,3 +1,4 @@
@@ -218,7 +209,7 @@ Index: deepnano/basecall_no_metrichor_devel.py
import argparse
from rnn_fin import RnnPredictor
import h5py
-@@ -76,17 +77,17 @@ def load_read_data(read_file):
+@@ -76,17 +77,17 @@
h5 = h5py.File(read_file, "r")
ret = {}
@@ -242,7 +233,7 @@ Index: deepnano/basecall_no_metrichor_devel.py
sampling_rate = h5["UniqueGlobalKey/channel_id"].attrs["sampling_rate"]
-@@ -95,7 +96,7 @@ def load_read_data(read_file):
+@@ -95,7 +96,7 @@
ret["called_complement"] = h5["Analyses/Basecall_2D_000/BaseCalled_complement/Fastq"][()].split('\n')[1]
ret["called_2d"] = h5["Analyses/Basecall_2D_000/BaseCalled_2D/Fastq"][()].split('\n')[1]
except Exception as e:
@@ -251,7 +242,7 @@ Index: deepnano/basecall_no_metrichor_devel.py
return None
events = base_events[temp_comp_loc["temp"][0]:temp_comp_loc["temp"][1]]
tscale2, tscale_sd2, tshift2 = get_scaling_template(events)
-@@ -203,18 +204,18 @@ if "all" in types or "2d" in types:
+@@ -203,18 +204,18 @@
assert do_template or do_complement or do_2d, "Nothing to do"
if do_template or do_2d:
@@ -276,7 +267,7 @@ Index: deepnano/basecall_no_metrichor_devel.py
chars = "ACGT"
mapping = {"A": 0, "C": 1, "G": 2, "T": 3, "N": 4}
-@@ -227,24 +228,24 @@ for i, read in enumerate(args.reads):
+@@ -227,24 +228,24 @@
if True:
data = load_read_data(read)
# except Exception as e:
@@ -310,7 +301,7 @@ Index: deepnano/basecall_no_metrichor_devel.py
for a, b in zip(o1m, o2m):
if a < 4:
fo.write(chars[a])
-@@ -255,7 +256,7 @@ for i, read in enumerate(args.reads):
+@@ -255,7 +256,7 @@
o1m = (np.argmax(o1, 1))
o2m = (np.argmax(o2, 1))
if do_template:
@@ -319,7 +310,7 @@ Index: deepnano/basecall_no_metrichor_devel.py
for a, b in zip(o1m, o2m):
if a < 4:
fo.write(chars[a])
-@@ -267,7 +268,7 @@ for i, read in enumerate(args.reads):
+@@ -267,7 +268,7 @@
o1c, o2c = comp_net.predict(data["comp_events"])
o1cm = (np.argmax(o1c, 1))
o2cm = (np.argmax(o2c, 1))
@@ -328,7 +319,7 @@ Index: deepnano/basecall_no_metrichor_devel.py
for a, b in zip(o1cm, o2cm):
if a < 4:
fo.write(chars[a])
-@@ -278,7 +279,7 @@ for i, read in enumerate(args.reads):
+@@ -278,7 +279,7 @@
o1cm = (np.argmax(o1c, 1))
o2cm = (np.argmax(o2c, 1))
if do_complement:
@@ -337,7 +328,7 @@ Index: deepnano/basecall_no_metrichor_devel.py
for a, b in zip(o1cm, o2cm):
if a < 4:
fo.write(chars[a])
-@@ -288,20 +289,20 @@ for i, read in enumerate(args.reads):
+@@ -288,20 +289,20 @@
if do_2d:
f2d = open("2d.in", "w")
@@ -356,8 +347,7 @@ Index: deepnano/basecall_no_metrichor_devel.py
+ print(" ".join(map(str, a)), file=f2d)
+ print(" ".join(map(str, b)), file=f2d)
f2d.close()
-- os.system("/usr/lib/deepnano/align_2d <2d.in >2d.out")
-+ os.system("./align_2d <2d.in >2d.out")
+ os.system("/usr/lib/deepnano/align_2d <2d.in >2d.out")
f2do = open("2d.out")
- call2d = f2do.next().strip()
- print >>fo, ">%d_2d_rnn_simple" % i
@@ -368,7 +358,7 @@ Index: deepnano/basecall_no_metrichor_devel.py
start_temp_ours = None
end_temp_ours = None
-@@ -330,7 +331,7 @@ for i, read in enumerate(args.reads):
+@@ -330,7 +331,7 @@
o1c, o2c = big_net.predict(events_2d)
o1cm = (np.argmax(o1c, 1))
o2cm = (np.argmax(o2c, 1))
@@ -377,7 +367,7 @@ Index: deepnano/basecall_no_metrichor_devel.py
for a, b in zip(o1cm, o2cm):
if a < 4:
fo.write(chars[a])
-@@ -340,7 +341,7 @@ for i, read in enumerate(args.reads):
+@@ -340,7 +341,7 @@
o1c, o2c = big_net.predict(data["2d_events"])
o1cm = (np.argmax(o1c, 1))
o2cm = (np.argmax(o2c, 1))
@@ -386,7 +376,7 @@ Index: deepnano/basecall_no_metrichor_devel.py
for a, b in zip(o1cm, o2cm):
if a < 4:
fo.write(chars[a])
-@@ -362,10 +363,10 @@ for i, read in enumerate(args.reads):
+@@ -362,10 +363,10 @@
end_comp_th = a[1]
start_comp_th = a[1]
@@ -404,8 +394,6 @@ Index: deepnano/basecall_no_metrichor_devel.py
+ print(start_temp_th, end_temp_th, start_comp_th, end_comp_th)
+ print(1. * len(data["al"]) / (end_temp_th - start_temp_th + end_comp_th - start_comp_th) )
+ print()
-Index: deepnano/helpers.py
-===================================================================
--- deepnano.orig/helpers.py
+++ deepnano/helpers.py
@@ -1,3 +1,4 @@
@@ -413,7 +401,7 @@ Index: deepnano/helpers.py
from rnn_fin import RnnPredictor
import h5py
import sys
-@@ -22,7 +23,7 @@ def predict_and_write(events, ntwk, fo,
+@@ -22,7 +23,7 @@
if fo:
o1m = (np.argmax(o1, 1))
o2m = (np.argmax(o2, 1))
@@ -422,8 +410,6 @@ Index: deepnano/helpers.py
for a, b in zip(o1m, o2m):
if a < 4:
fo.write(chars[a])
-Index: deepnano/r9/basecall.py
-===================================================================
--- deepnano.orig/r9/basecall.py
+++ deepnano/r9/basecall.py
@@ -1,3 +1,4 @@
@@ -431,7 +417,7 @@ Index: deepnano/r9/basecall.py
from rnnf import Rnn
from qrnnf import Rnn as Rnnq
import h5py
-@@ -65,12 +66,12 @@ def basecall(filename, output_file):
+@@ -65,12 +66,12 @@
h5 = h5py.File(filename, "r")
events = get_events(h5)
if events is None:
@@ -446,7 +432,7 @@ Index: deepnano/r9/basecall.py
h5.close()
return 0
-@@ -106,14 +107,14 @@ def basecall(filename, output_file):
+@@ -106,14 +107,14 @@
om = np.vstack((o1m,o2m)).reshape((-1,),order='F')
output = "".join(map(lambda x: alph[x], om)).replace("N", "")
@@ -464,7 +450,7 @@ Index: deepnano/r9/basecall.py
return 0
alph = "ACGTN"
-@@ -156,7 +157,7 @@ if len(args.reads) or len(args.directory
+@@ -156,7 +157,7 @@
if args.debug:
total_events += current_events
time_diff = (datetime.datetime.now() - start_time).total_seconds() + 0.000001
@@ -473,7 +459,7 @@ Index: deepnano/r9/basecall.py
fo.close()
-@@ -165,14 +166,14 @@ if len(args.watch) != 0:
+@@ -165,14 +166,14 @@
from watchdog.observers import Observer
from watchdog.events import PatternMatchingEventHandler
except:
@@ -490,11 +476,9 @@ Index: deepnano/r9/basecall.py
file_name = str(os.path.basename(event.src_path))
fasta_file_name = os.path.splitext(event.src_path)[0] + '.fasta'
with open(fasta_file_name, "w") as fo:
-Index: deepnano/r9/extract_events.py
-===================================================================
--- deepnano.orig/r9/extract_events.py
+++ deepnano/r9/extract_events.py
-@@ -18,7 +18,7 @@ defs = {
+@@ -18,7 +18,7 @@
}
def get_raw(h5):
@@ -503,7 +487,7 @@ Index: deepnano/r9/extract_events.py
raw = h5["Raw/Reads"][rk]["Signal"]
meta = h5["UniqueGlobalKey/channel_id"].attrs
-@@ -69,7 +69,7 @@ def get_tstat(s, s2, wl):
+@@ -69,7 +69,7 @@
def extract_events(h5, chem):
@@ -512,8 +496,6 @@ Index: deepnano/r9/extract_events.py
raw, sl = get_raw(h5)
events = event_detect(raw, sl, **defs[chem]["ed_params"])
-Index: deepnano/r9/training/realign.py
-===================================================================
--- deepnano.orig/r9/training/realign.py
+++ deepnano/r9/training/realign.py
@@ -1,3 +1,4 @@
@@ -521,7 +503,7 @@ Index: deepnano/r9/training/realign.py
from qrnn import BatchNet
import pickle
import sys
-@@ -53,17 +54,17 @@ def realign(s):
+@@ -53,17 +54,17 @@
# fo.write(alph[b])
# fo.close()
f = open(base_dir+"tmpb-%s.in" % s, "w")
@@ -544,7 +526,7 @@ Index: deepnano/r9/training/realign.py
return
f = open(base_dir+"tmpb-%s.out" % s)
-@@ -72,9 +73,9 @@ def realign(s):
+@@ -72,9 +73,9 @@
data_y2[ps][i] = mapping[l[1]]
fo = open(names[s] + "r", "w")
@@ -556,7 +538,7 @@ Index: deepnano/r9/training/realign.py
fo.close()
if __name__ == '__main__':
-@@ -95,14 +96,14 @@ if __name__ == '__main__':
+@@ -95,14 +96,14 @@
f = open(fn)
ref = f.readline()
if len(ref) > 30000:
@@ -573,8 +555,6 @@ Index: deepnano/r9/training/realign.py
sys.stdout.flush()
for l in f:
its = l.strip().split()
-Index: deepnano/r9/training/theano_toolkit/parameters.py
-===================================================================
--- deepnano.orig/r9/training/theano_toolkit/parameters.py
+++ deepnano/r9/training/theano_toolkit/parameters.py
@@ -1,6 +1,7 @@
@@ -586,7 +566,7 @@ Index: deepnano/r9/training/theano_toolkit/parameters.py
from functools import reduce
import inspect
-@@ -24,7 +25,7 @@ class Parameters():
+@@ -24,7 +25,7 @@
name=name
)
else:
@@ -595,7 +575,7 @@ Index: deepnano/r9/training/theano_toolkit/parameters.py
params[name].set_value(np.asarray(
array,
-@@ -60,7 +61,7 @@ class Parameters():
+@@ -60,7 +61,7 @@
if k in loaded:
params[k].set_value(loaded[k])
else:
@@ -604,8 +584,6 @@ Index: deepnano/r9/training/theano_toolkit/parameters.py
def __enter__(self):
_, _, _, env_locals = inspect.getargvalues(
-Index: deepnano/r9/training/theano_toolkit/updates.py
-===================================================================
--- deepnano.orig/r9/training/theano_toolkit/updates.py
+++ deepnano/r9/training/theano_toolkit/updates.py
@@ -1,4 +1,4 @@
@@ -614,7 +592,7 @@ Index: deepnano/r9/training/theano_toolkit/updates.py
import theano.tensor as T
import numpy as np
from parameters import Parameters
-@@ -15,7 +15,7 @@ def nan_shield(parameters, deltas, other
+@@ -15,7 +15,7 @@
delta_sum = sum(T.sum(d) for d in deltas)
not_finite = T.isnan(delta_sum) | T.isinf(delta_sum)
parameter_updates = [(p, T.switch(not_finite, 0.9 * p, p - d))
@@ -623,7 +601,7 @@ Index: deepnano/r9/training/theano_toolkit/updates.py
other_updates = [(p, T.switch(not_finite, p, u))
for p, u in other_updates]
return parameter_updates, other_updates
-@@ -51,25 +51,25 @@ def adadelta(parameters, gradients,
+@@ -51,25 +51,25 @@
shapes = get_shapes(parameters)
acc_gradients_sq = [create_param(P, "grad_sq_" + p.name, np.zeros(s))
@@ -656,7 +634,7 @@ Index: deepnano/r9/training/theano_toolkit/updates.py
gradient_sq_updates = zip(acc_gradients_sq, gradients_sq_new)
deltas_sq_updates = zip(acc_deltas_sq, deltas_sq_new)
-@@ -81,11 +81,11 @@ def adagrad(parameters, gradients, learn
+@@ -81,11 +81,11 @@
shapes = get_shapes(parameters)
grad_sq = [create_param(P, "acc_sq_" + p.name, np.zeros(s))
@@ -671,7 +649,7 @@ Index: deepnano/r9/training/theano_toolkit/updates.py
grad_sq_update = zip(grad_sq, grad_sq_new)
return deltas, grad_sq_update
-@@ -99,7 +99,7 @@ def momentum(parameters, gradients, mu=0
+@@ -99,7 +99,7 @@
mu = m * (1 - 3.0 / (P.t + 5)) + (1 - m) * mu
shapes = get_shapes(parameters)
deltas = [create_param(P, "deltas_" + p.name, np.zeros(s))
@@ -680,7 +658,7 @@ Index: deepnano/r9/training/theano_toolkit/updates.py
delta_nexts = [mu * delta + eps * grad for delta,
grad in zip(deltas, gradients)]
delta_updates = [(delta, delta_next)
-@@ -116,24 +116,24 @@ def rmsprop(parameters, gradients,
+@@ -116,24 +116,24 @@
P=None):
shapes = get_shapes(parameters)
sq_acc = [create_param(P, "sq_acc_" + p.name, np.zeros(s))
@@ -715,7 +693,7 @@ Index: deepnano/r9/training/theano_toolkit/updates.py
return deltas, acc_updates + sq_acc_updates + delta_updates
-@@ -149,15 +149,15 @@ def adam(parameters, gradients,
+@@ -149,15 +149,15 @@
P.t = np.float32(1)
moment1_acc = [create_param(P, "moment1_" + p.name, np.zeros(s))
@@ -734,8 +712,6 @@ Index: deepnano/r9/training/theano_toolkit/updates.py
new_m1 = moment1_decay * m1 + (1 - moment1_decay) * g
new_m2 = moment2_decay * m2 + (1 - moment2_decay) * T.sqr(g)
bc_m1 = new_m1 / (1 - moment1_decay**P.t)
-Index: deepnano/r9/training/train.py
-===================================================================
--- deepnano.orig/r9/training/train.py
+++ deepnano/r9/training/train.py
@@ -1,3 +1,4 @@
@@ -743,7 +719,7 @@ Index: deepnano/r9/training/train.py
from qrnn import BatchNet
import pickle
import sys
-@@ -55,16 +56,16 @@ def realign(s):
+@@ -55,16 +56,16 @@
# fo.close()
f = open(base_dir+"tmpb-%s.in" % s, "w")
lc = 0
@@ -765,7 +741,7 @@ Index: deepnano/r9/training/train.py
sys.exit()
f = open(base_dir+"tmpb-%s.out" % s)
-@@ -73,9 +74,9 @@ def realign(s):
+@@ -73,9 +74,9 @@
data_y2[ps][i] = mapping[l[1]]
fo = open(names[s] + "r", "w")
@@ -777,7 +753,7 @@ Index: deepnano/r9/training/train.py
fo.close()
return data_y[ps], data_y2[ps]
-@@ -98,7 +99,7 @@ if __name__ == '__main__':
+@@ -98,7 +99,7 @@
f = open(fn)
ref = f.readline()
if len(ref) > 30000:
@@ -786,7 +762,7 @@ Index: deepnano/r9/training/train.py
continue
X = []
Y = []
-@@ -111,12 +112,12 @@ if __name__ == '__main__':
+@@ -111,12 +112,12 @@
Y.append(mapping[its[4][0]])
Y2.append(mapping[its[4][1]])
except:
@@ -801,7 +777,7 @@ Index: deepnano/r9/training/train.py
sys.stdout.flush()
refs.append(ref.strip())
names.append(fn)
-@@ -124,7 +125,7 @@ if __name__ == '__main__':
+@@ -124,7 +125,7 @@
data_y.append(np.array(Y, dtype=np.int32))
data_y2.append(np.array(Y2, dtype=np.int32))
@@ -810,7 +786,7 @@ Index: deepnano/r9/training/train.py
print ("done", sum(len(x) for x in refs), sum(len(x) for x in data_x))
sys.stdout.flush()
-@@ -153,7 +154,7 @@ if __name__ == '__main__':
+@@ -153,7 +154,7 @@
base_dir += "/"
batch_size = 8
n_batches = len(data_x) / batch_size + 1
@@ -819,7 +795,7 @@ Index: deepnano/r9/training/train.py
for epoch in range(1000):
taken_gc = []
-@@ -226,19 +227,19 @@ if __name__ == '__main__':
+@@ -226,19 +227,19 @@
sys.stdout.write('\r%d %f %f %f' % (s, tc / (s+1), tcs / max(1, ccs), tcb / max(1, ccb)))
sys.stdout.flush()
@@ -844,8 +820,6 @@ Index: deepnano/r9/training/train.py
sys.stdout.flush()
if epoch % 20 == 19:
-Index: deepnano/training/prepare_dataset.py
-===================================================================
--- deepnano.orig/training/prepare_dataset.py
+++ deepnano/training/prepare_dataset.py
@@ -1,3 +1,5 @@
@@ -854,7 +828,7 @@ Index: deepnano/training/prepare_dataset.py
import argparse
import os
import h5py
-@@ -18,7 +20,7 @@ for i, l in enumerate(finput):
+@@ -18,7 +20,7 @@
h5 = h5py.File(filename, "r")
fo = open(os.path.join(args.output_directory, "%s.txt" % i), "w")
@@ -863,7 +837,7 @@ Index: deepnano/training/prepare_dataset.py
base_loc = get_base_loc(h5)
if args.type == 'temp':
scale, scale_sd, shift, drift = extract_scaling(h5, "template", base_loc)
-@@ -29,14 +31,14 @@ for i, l in enumerate(finput):
+@@ -29,14 +31,14 @@
mean = (e["mean"] - shift) / scale
stdv = e["stdv"] / scale_sd
length = e["length"]
@@ -882,7 +856,7 @@ Index: deepnano/training/prepare_dataset.py
if args.type == 'comp':
scale, scale_sd, shift, drift = extract_scaling(h5, "complement", base_loc)
events = h5[base_loc+"/BaseCalled_%s/Events" % "complement"]
-@@ -46,14 +48,14 @@ for i, l in enumerate(finput):
+@@ -46,14 +48,14 @@
mean = (e["mean"] - shift) / scale
stdv = e["stdv"] / scale_sd
length = e["length"]
@@ -901,7 +875,7 @@ Index: deepnano/training/prepare_dataset.py
if args.type == '2d':
tscale, tscale_sd, tshift, tdrift = extract_scaling(h5, "template", base_loc)
cscale, cscale_sd, cshift, cdrift = extract_scaling(h5, "complement", base_loc)
-@@ -79,13 +81,13 @@ for i, l in enumerate(finput):
+@@ -79,13 +81,13 @@
stdv = e["stdv"] / cscale_sd
length = e["length"]
ev += [1] + preproc_event(mean, stdv, length)
@@ -919,8 +893,6 @@ Index: deepnano/training/prepare_dataset.py
fo.close()
-Index: deepnano/training/train.py
-===================================================================
--- deepnano.orig/training/train.py
+++ deepnano/training/train.py
@@ -1,3 +1,4 @@
@@ -928,7 +900,7 @@ Index: deepnano/training/train.py
from rnn import Rnn
import pickle
import sys
-@@ -25,15 +26,15 @@ def realign(s):
+@@ -25,15 +26,15 @@
o2m = (np.argmax(o2, 1))
alph = "ACGTN"
f = open(base_dir+"tmpb-%s.in" % s, "w")
@@ -950,7 +922,7 @@ Index: deepnano/training/train.py
sys.exit()
f = open(base_dir+"tmpb-%s.out" % s)
-@@ -60,7 +61,7 @@ if __name__ == '__main__':
+@@ -60,7 +61,7 @@
f = open(fn)
ref = f.readline()
if len(ref) > 30000:
@@ -959,7 +931,7 @@ Index: deepnano/training/train.py
continue
refs.append(ref.strip())
names.append(fn)
-@@ -76,12 +77,12 @@ if __name__ == '__main__':
+@@ -76,12 +77,12 @@
data_y.append(np.array(Y, dtype=np.int32))
data_y2.append(np.array(Y2, dtype=np.int32))
@@ -974,7 +946,7 @@ Index: deepnano/training/train.py
s_arr = []
p_arr = []
-@@ -101,7 +102,7 @@ if __name__ == '__main__':
+@@ -101,7 +102,7 @@
base_dir += "/"
batch_size = 1
n_batches = len(data_x) / batch_size
@@ -983,7 +955,7 @@ Index: deepnano/training/train.py
for epoch in range(1000):
if (epoch % 20 == 0 and epoch > 0) or (epoch == 0):
-@@ -151,14 +152,14 @@ if __name__ == '__main__':
+@@ -151,14 +152,14 @@
sys.stdout.write('\r%d' % s)
sys.stdout.flush()
=====================================
debian/patches/series
=====================================
@@ -1,3 +1,5 @@
she_bang.patch
path_align_2d.patch
py3-compat.patch
+fix-TypeError.patch
+basecall_no_metrichor.patch
=====================================
debian/salsa-ci.yml
=====================================
@@ -0,0 +1,4 @@
+---
+include:
+ - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/salsa-ci.yml
+ - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/pipeline-jobs.yml
=====================================
debian/tests/run-test.sh
=====================================
@@ -3,12 +3,17 @@
pkg="deepnano"
test_required_pkg="${pkg}-data"
datadir=/usr/share/poretools/data
+# Fail earlier than 2:17 in case #921566 still applies; each test has this
+# amount of seconds to run.
+timeout=600
if [ "$AUTOPKGTEST_TMP" = "" ] ; then
AUTOPKGTEST_TMP=$(mktemp -d /tmp/${pkg}-test.XXXXXX)
trap "rm -rf $AUTOPKGTEST_TMP" 0 INT QUIT ABRT PIPE TERM
fi
+HOME="$AUTOPKGTEST_TMP"
+export HOME
cp -a /usr/share/${test_required_pkg}/* $AUTOPKGTEST_TMP
mkdir -p $AUTOPKGTEST_TMP/test_data/
@@ -18,13 +23,19 @@ for d in `ls $datadir | sort | head -n10` ; do cp $datadir/$d $AUTOPKGTEST_TMP/t
cd $AUTOPKGTEST_TMP
find . -name "*gz" -exec gunzip \{\} \;
-#echo '!!!!!!!!!!'
-#ls -R
-echo -e "\n#1 - deepnano_basecall"
-OMP_NUM_THREADS=`nproc` deepnano_basecall test_data/*
+
+# Cap parallelization level: may help avoid #921566.
+if [ "$(nproc)" -lt 16 ]
+then OMP_NUM_THREADS="$(nproc)"
+else OMP_NUM_THREADS=16
+fi
+export OMP_NUM_THREADS
+
+echo "\n#1 - deepnano_basecall"
+timeout --verbose "$timeout" deepnano_basecall test_data/*
cat output.fasta
-echo -e "\n#2 - deepnano_basecall_no_metrichor"
-OMP_NUM_THREADS=`nproc` deepnano_basecall_no_metrichor test_data/*
+echo "\n#2 - deepnano_basecall_no_metrichor"
+timeout --verbose "$timeout" deepnano_basecall_no_metrichor test_data/*
echo "PASS"
=====================================
debian/upstream/metadata
=====================================
@@ -17,3 +17,4 @@ Registry:
Entry: NA
- Name: SciCrunch
Entry: NA
+Repository: https://bitbucket.org/vboza/deepnano.git
=====================================
debian/watch
=====================================
@@ -3,6 +3,6 @@ version=4
opts="mode=git,pretty=0.0+git%cd.%h" \
https://bitbucket.org/vboza/deepnano.git HEAD
-# version=3
+# version=4
# upstream has not proper versioning
View it on GitLab: https://salsa.debian.org/med-team/deepnano/-/compare/9473e9c3befe446250b5270aae1065f1cdc2bce8...4da728bff97cae7d44411508f76c590ed6ee11ab
--
View it on GitLab: https://salsa.debian.org/med-team/deepnano/-/compare/9473e9c3befe446250b5270aae1065f1cdc2bce8...4da728bff97cae7d44411508f76c590ed6ee11ab
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/debian-med-commit/attachments/20221218/a6790007/attachment-0001.htm>
More information about the debian-med-commit
mailing list