[Git][security-tracker-team/security-tracker][master] 31 commits: Force keys() to get evaluated as list
Emilio Pozuelo Monfort
pochu at debian.org
Wed Jul 29 09:22:27 BST 2020
Emilio Pozuelo Monfort pushed to branch master at Debian Security Tracker / security-tracker
Commits:
e6904885 by Brian May at 2020-07-29T10:12:59+02:00
Force keys() to get evaluated as list
Under Python 3, keys() doesn't return a list (as in Python 2), but
must be evaluated before it can be used as a list.
- - - - -
6b1ca897 by Brian May at 2020-07-29T10:12:59+02:00
Force filter() to get evaluated as list
Under Python 3, filter() doesn't return a list (as in Python 2), but
must be evaluated before it can be used as a list.
- - - - -
6e7da212 by Brian May at 2020-07-29T10:12:59+02:00
Force map() to get evaluated as list
Under Python 3, map() doesn't return a list (as in Python 2), but
must be evaluated before it can be used as a list.
- - - - -
41452daf by Brian May at 2020-07-29T10:12:59+02:00
Use urllib urllib instead of urllib2
- - - - -
52d8f81e by Brian May at 2020-07-29T10:12:59+02:00
Replace cPickle with pickle library
cPickle isn't available in Python 3.
- - - - -
98dbd1d4 by Emilio Pozuelo Monfort at 2020-07-29T10:12:59+02:00
config.py: force keys() to get evaluated as list
- - - - -
37e775cc by Emilio Pozuelo Monfort at 2020-07-29T10:20:33+02:00
lib/python: use isinstance rather than types
types.TypeFoo are gone in python3.
- - - - -
df9081c7 by Emilio Pozuelo Monfort at 2020-07-29T10:20:41+02:00
bugs.py: encode the string before using the digest
Needed for py3 compatibility.
- - - - -
4f4b39aa by Emilio Pozuelo Monfort at 2020-07-29T10:20:41+02:00
tracker_service.py: fix use of tabs
- - - - -
62a41db3 by Emilio Pozuelo Monfort at 2020-07-29T10:20:41+02:00
test-web-server: call tracker_service.py directly
It's an executable, so no need to call python first and potentially
use a different interpreter than what tracker_service.py declares
in the hashbang.
- - - - -
83f4314d by Emilio Pozuelo Monfort at 2020-07-29T10:20:41+02:00
debian_support: use BytesIO for GzipFile in python3
- - - - -
9a15667d by Emilio Pozuelo Monfort at 2020-07-29T10:20:41+02:00
web_support.py: fix octal numbers under py3
- - - - -
2a122964 by Emilio Pozuelo Monfort at 2020-07-29T10:20:41+02:00
lib/python: replace tabs with spaces
- - - - -
98f3af78 by Emilio Pozuelo Monfort at 2020-07-29T10:20:41+02:00
web_support.py: adapt StringIO for python3
- - - - -
3d659f13 by Emilio Pozuelo Monfort at 2020-07-29T10:20:41+02:00
web_support.py: adapt HTTPServer imports for python3
- - - - -
95f779f1 by Emilio Pozuelo Monfort at 2020-07-29T10:20:41+02:00
web_support.py: decode data when necessary
We can't pass bytes under python3
- - - - -
6af514f6 by Emilio Pozuelo Monfort at 2020-07-29T10:20:41+02:00
web_support.py: adapt urllib.quote for python3
- - - - -
c7cd4e77 by Emilio Pozuelo Monfort at 2020-07-29T10:20:41+02:00
web_support.py: make_list: don't subscript the list
We may get objects that are not subscriptable in py3, such as
map objects, so just iterate over the whole object and pop the
last separator instead.
- - - - -
271295dd by Emilio Pozuelo Monfort at 2020-07-29T10:20:41+02:00
tracker_service.py: use a lambda function to sort
As cmp is gone in py3. Also don't pass it as a positional argument.
- - - - -
a922e608 by Emilio Pozuelo Monfort at 2020-07-29T10:20:41+02:00
tracker_service.py: open the binary image file as binary
- - - - -
6a8f2e84 by Emilio Pozuelo Monfort at 2020-07-29T10:20:41+02:00
tracker_service.py: evaluate dict items as a list
In python3 this is an iterator, which breaks when the dict
is modified. To avoid that, force it to be evaluated as a
list.
- - - - -
40bea495 by Emilio Pozuelo Monfort at 2020-07-29T10:20:41+02:00
debian_support.py: add py3 compatibility for apt-update-file
- - - - -
ef4aa74d by Emilio Pozuelo Monfort at 2020-07-29T10:20:41+02:00
security_db: use pickle's dumps and loads
Rather than using StringIO in py2 and BytesIO in py3 and porting
away from buffer which is also gone in py3, let's just use dumps
and loads and let the pickle library handle the memory representation
for us.
- - - - -
751b43a8 by Emilio Pozuelo Monfort at 2020-07-29T10:20:41+02:00
security_db: encode data before passing it to base64
In python3, base64 takes bytes.
- - - - -
590a1fbf by Emilio Pozuelo Monfort at 2020-07-29T10:20:41+02:00
debian_support: decode lines when necessary
We sometimes get passed lines as bytes, which we need to decode
under python3.
We should probably add an argument to PackageFile's constructor
for when we get a fileObj argument, but let's do that when we
no longer have to worry about py2 and py3 compatibility.
- - - - -
d1cf4a37 by Emilio Pozuelo Monfort at 2020-07-29T10:20:41+02:00
bugs.py: sort using a lambda key function
- - - - -
8bf2fb17 by Emilio Pozuelo Monfort at 2020-07-29T10:20:41+02:00
bugs.py: make PackageNoteNoDSA.release a Release object
Like in the other PackageNote objects
- - - - -
4adb5a93 by Emilio Pozuelo Monfort at 2020-07-29T10:20:41+02:00
bugs.py: sort using Release's sort
string sort doesn't work here, as buster < jessie < stretch.
However Release's sort will dtrt.
- - - - -
4337fae9 by Emilio Pozuelo Monfort at 2020-07-29T10:20:41+02:00
tracker_service: properly sort fixed versions table
- - - - -
59eee919 by Brian May at 2020-07-29T10:20:41+02:00
Add comparison functions required for Python3
- - - - -
dd72ac01 by Emilio Pozuelo Monfort at 2020-07-29T10:20:41+02:00
debian_support: remove PseudoEnum.__cmp__
It uses cmp, which is no longer available in python3. But that's
fine, as we are now comparing using pure key functions. So let's
remove the cmp helper rather then reintroducing cmp().
- - - - -
10 changed files:
- bin/src2bin_text.py
- bin/test-web-server
- bin/tracker_service.py
- lib/python/bugs.py
- lib/python/config.py
- lib/python/debian_support.py
- + lib/python/helpers.py
- lib/python/sectracker/xpickle.py
- lib/python/security_db.py
- lib/python/web_support.py
Changes:
=====================================
bin/src2bin_text.py
=====================================
@@ -24,7 +24,7 @@ def joinEN(words):
def filterPkg(bins,rms):
for rm in rms:
bins = filter(lambda x: not x.endswith('-%s' % rm), bins)
- return bins
+ return list(bins)
def getBin(srcPkg):
bins = soappy_query(default_url,'binary_names',source=srcPkg)
=====================================
bin/test-web-server
=====================================
@@ -4,7 +4,7 @@ set -e
server_port=10605
ip_address="127.0.0.1"
-service=tracker_service.py
+service=./tracker_service.py
url="http://$ip_address:$server_port/tracker"
bindir="`dirname $0`"
@@ -15,4 +15,4 @@ fi
cd "$bindir"
echo "URL: $url"
-python "$service" "$url" "$ip_address" "$server_port" ../data/security.db || true
+"$service" "$url" "$ip_address" "$server_port" ../data/security.db || true
=====================================
bin/tracker_service.py
=====================================
@@ -5,6 +5,7 @@ sys.path.insert(0,'../lib/python')
import bugs
import config
import re
+import debian_support
import security_db
from web_support import *
import json
@@ -37,7 +38,7 @@ def clean_dict(d):
This alters the input so you may wish to ``copy`` the dict first.
"""
# d.iteritems isn't used as you can't del or the iterator breaks.
- for key, value in d.items():
+ for key, value in list(d.items()):
if value is None:
del d[key]
elif isinstance(value, dict):
@@ -107,10 +108,10 @@ class BugFilter:
return filterlow or filtermed or filterhigh or filterund or filteruni or filteruna or filterend
def remoteFiltered(self, remote):
- filterr = self.params['remote'] and remote and remote is not None
- filterl = self.params['local'] and not remote and remote is not None
- filteru = self.params['unclear'] and remote is None
- return filterr or filterl or filteru
+ filterr = self.params['remote'] and remote and remote is not None
+ filterl = self.params['local'] and not remote and remote is not None
+ filteru = self.params['unclear'] and remote is None
+ return filterr or filterl or filteru
def nodsaFiltered(self, nodsa):
"""Returns True for no DSA issues if filtered."""
@@ -186,14 +187,14 @@ class TrackerService(webservice_base_class):
def page_style_css(self, path, params, url):
f=open('../static/style.css', 'r')
- content=f.read()
- f.close()
+ content=f.read()
+ f.close()
return BinaryResult(content,'text/css')
def page_logo_png(self, path, params, url):
- f=open('../static/logo.png', 'r')
- content=f.read()
- f.close()
+ f=open('../static/logo.png', 'rb')
+ content=f.read()
+ f.close()
return BinaryResult(content,'image/png')
def page_distributions_json(self, path, params, url):
@@ -486,15 +487,15 @@ data source.""")],
def gen_data():
notes_sorted = bug.notes[:]
- notes_sorted.sort(lambda a, b: cmp(a.package, b.package))
+ notes_sorted.sort(key=lambda n: (n.package, n.release or debian_support.internRelease('sid')))
for n in notes_sorted:
if n.release:
rel = str(n.release)
else:
rel = '(unstable)'
urgency = str(n.urgency)
- if urgency == 'end-of-life':
- urgency = self.make_purple('end-of-life')
+ if urgency == 'end-of-life':
+ urgency = self.make_purple('end-of-life')
if n.fixed_version:
ver = str(n.fixed_version)
if ver == '0':
@@ -1470,9 +1471,9 @@ Debian bug number.'''),
append(FOOTER(P(A(url.scriptRelative(""), "Home"),
" - ", A(url.absolute("https://www.debian.org/security/"),
"Debian Security"),
- " - ", A(url.absolute("https://salsa.debian.org/security-tracker-team/security-tracker/blob/master/bin/tracker_service.py"),
- "Source"),
- " ", A(url.absolute("https://salsa.debian.org/security-tracker-team/security-tracker"), "(Git)"),
+ " - ", A(url.absolute("https://salsa.debian.org/security-tracker-team/security-tracker/blob/master/bin/tracker_service.py"),
+ "Source"),
+ " ", A(url.absolute("https://salsa.debian.org/security-tracker-team/security-tracker"), "(Git)"),
)))
if search_in_page:
on_load = "selectSearch()"
@@ -1729,7 +1730,7 @@ Debian bug number.'''),
return SPAN(contents, _class="yellow")
def make_purple(self, contents):
- return SPAN(contents, _class="purple")
+ return SPAN(contents, _class="purple")
def make_green(self, contents):
return SPAN(contents, _class="green")
=====================================
lib/python/bugs.py
=====================================
@@ -19,9 +19,10 @@ import debian_support
import functools
import os
import re
-import types
import hashlib
+from helpers import isstring
+
class Urgency(debian_support.PseudoEnum): pass
def listUrgencies():
@@ -55,20 +56,19 @@ class PackageNote:
def __init__(self, package, fixed_version, release, urgency):
self.id = None
self.package = package
- if (fixed_version is not None
- and type(fixed_version) in types.StringTypes):
+ if (isstring(fixed_version)):
self.fixed_version = debian_support.Version(fixed_version)
else:
self.fixed_version = fixed_version
if release == '':
self.release = None
else:
- if type(release) == types.StringType:
+ if isstring(release):
release = debian_support.internRelease(release)
if release is None:
raise ValueError("invalid release")
self.release = release
- if type(urgency) == types.StringType:
+ if isstring(urgency):
urgency = internUrgency(urgency)
if urgency is None:
raise ValueError("invalid urgency")
@@ -166,14 +166,17 @@ class PackageNoteParsed(PackageNote):
class PackageNoteNoDSA:
def __init__(self, package, release, comment, reason=None):
- assert type(package) == types.StringType and package != ''
- assert type(release) == types.StringType and release != ''
- assert type(comment) == types.StringType
+ assert isstring(package) and package != ''
+ assert isstring(release) and release != ''
+ assert isstring(comment)
if not reason:
reason = ''
else:
- assert type(reason) == types.StringType
+ assert isstring(reason)
self.package = package
+ release = debian_support.internRelease(release)
+ if release is None:
+ raise ValueError("invalid release")
self.release = release
self.comment = comment
self.reason = reason
@@ -182,7 +185,7 @@ class PackageNoteNoDSA:
cursor.execute("""INSERT INTO package_notes_nodsa
(bug_name, package, release, comment, reason)
VALUES (?, ?, ?, ?, ?)""",
- (bug_name, self.package, self.release,
+ (bug_name, self.package, str(self.release),
self.comment, self.reason))
class BugBase:
@@ -191,7 +194,7 @@ class BugBase:
re_cve_name = re.compile(r'^CVE-\d{4}-\d{4,}$')
def __init__(self, fname, lineno, date, name, description, comments):
- assert type(fname) in types.StringTypes
+ assert isstring(fname)
lineno = to_integer(lineno)
self.source_file = fname
self.source_line = lineno
@@ -266,8 +269,8 @@ class Bug(BugBase):
for n in notes:
assert isinstance(n, PackageNote) \
or isinstance(n, PackageNoteNoDSA)
- assert len(xref) == 0 or type(xref[0]) == types.StringType
- assert type(not_for_us) == types.BooleanType
+ assert len(xref) == 0 or isstring(xref[0])
+ assert isinstance(not_for_us, bool)
BugBase.__init__(self, fname, lineno, date, name,
description, comments)
self.notes = notes
@@ -287,19 +290,12 @@ class Bug(BugBase):
notes[key].merge(n)
else:
notes[key] = n
- l = notes.keys()
+ l = list(notes.keys())
# The release part of a key can be None, so we have to deal
# with that when sorting.
- def compare(a, b):
- r = cmp(a[0], b[0])
- if r:
- return r
- ar = str(a[1] or '')
- br = str(b[1] or '')
- return cmp(ar, br)
- l.sort(key=functools.cmp_to_key(compare))
-
+ l.sort(key=lambda n: (n[0], n[1] or debian_support.internRelease('sid')))
+
nts = []
for key in l:
nts.append(notes[key])
@@ -307,7 +303,7 @@ class Bug(BugBase):
class BugFromDB(Bug):
def __init__(self, cursor, name):
- assert type(name) in types.StringTypes
+ assert isstring(name)
def lookup(bug):
for r in cursor.execute('SELECT * FROM bugs WHERE name = ?',
@@ -407,7 +403,7 @@ def temp_bug_name(bug_number, description):
"""Build a unique temporary name from the bug number and a
truncated hash of the description."""
digest = hashlib.md5()
- digest.update(description)
+ digest.update(description.encode('utf-8'))
hexdigest = digest.hexdigest()[0:6].upper()
return 'TEMP-%07d-%s' % (bug_number, hexdigest)
=====================================
lib/python/config.py
=====================================
@@ -42,7 +42,7 @@ def get_supported_releases():
def get_all_releases():
config = get_config()
- return config.keys()
+ return list(config.keys())
def get_release_codename(release, suffix=''):
config = get_config()
=====================================
lib/python/debian_support.py
=====================================
@@ -19,14 +19,24 @@ from __future__ import print_function
"""This module implements facilities to deal with Debian-specific metadata."""
import gzip
+import io
import json
import os.path
import re
import sys
import tempfile
-import types
-import urllib2
-from cStringIO import StringIO
+
+try:
+ from urllib.request import urlopen
+except ImportError:
+ from urllib2 import urlopen
+
+try:
+ from cStringIO import StringIO as streamIO
+except ImportError:
+ from io import BytesIO as streamIO
+
+from helpers import isstring
try:
from hashlib import sha1
@@ -54,7 +64,7 @@ class ParseError(Exception):
"""
def __init__(self, filename, lineno, msg):
- assert type(lineno) == types.IntType
+ assert isinstance(lineno, int)
self.filename = filename
self.lineno = lineno
self.msg = msg
@@ -83,11 +93,13 @@ class Version:
def __init__(self, version):
"""Creates a new Version object."""
- t = type(version)
- if t == types.UnicodeType:
- version = version.encode('UTF-8')
- else:
- assert t == types.StringType, repr(version)
+ try:
+ if isinstance(version, unicode):
+ version = version.encode('UTF-8')
+ except:
+ pass
+
+ assert isstring(version), repr(version)
assert version != ""
self.__asString = version
self.__forCompare = _version_normalize_regexp.sub("", version)
@@ -104,6 +116,21 @@ class Version:
except AttributeError:
return apt_pkg.VersionCompare(self.__forCompare, other.__forCompare)
+ def __lt__(self, other):
+ return self.__cmp__(other) < 0
+
+ def __le__(self, other):
+ return self.__cmp__(other) <= 0
+
+ def __eq__(self, other):
+ return self.__cmp__(other) == 0
+
+ def __gt__(self, other):
+ return self.__cmp__(other) > 0
+
+ def __ge__(self, other):
+ return self.__cmp__(other) >= 0
+
def version_compare(a, b):
"""Compares two versions according to the Debian algorithm.
@@ -138,8 +165,16 @@ class PackageFile:
self.file = fileObj
self.lineno = 0
- def __iter__(self):
+ def readline(self):
line = self.file.readline()
+
+ if line != None and not isstring(line):
+ line = line.decode('utf-8')
+
+ return line
+
+ def __iter__(self):
+ line = self.readline()
self.lineno += 1
pkg = []
while line:
@@ -148,7 +183,7 @@ class PackageFile:
self.raiseSyntaxError('expected package record')
yield pkg
pkg = []
- line = self.file.readline()
+ line = self.readline()
self.lineno += 1
continue
@@ -159,7 +194,7 @@ class PackageFile:
contents = contents or ''
while True:
- line = self.file.readline()
+ line = self.readline()
self.lineno += 1
match = self.re_continuation.match(line)
if match:
@@ -187,10 +222,18 @@ class PseudoEnum:
return '%s(%r)'% (self.__class__.__name__, self._name)
def __str__(self):
return self._name
- def __cmp__(self, other):
- return cmp(self._order, other._order)
def __hash__(self):
return hash(self._order)
+ def __lt__(self, other):
+ return self._order < other._order
+ def __le__(self, other):
+ return self._order <= other._order
+ def __eq__(self, other):
+ return self._order == other._order
+ def __gt__(self, other):
+ return self._order > other._order
+ def __ge__(self, other):
+ return self._order >= other._order
class Release(PseudoEnum): pass
@@ -211,6 +254,8 @@ del listReleases
def readLinesSHA1(lines):
m = sha1()
for l in lines:
+ if sys.version_info.major == 3:
+ l = l.encode('utf-8')
m.update(l)
return m.hexdigest()
@@ -291,11 +336,14 @@ def downloadGunzipLines(remote):
Returns the lines in the file."""
- data = urllib2.urlopen(remote, timeout=TIMEOUT)
+ data = urlopen(remote, timeout=TIMEOUT)
try:
- gfile = gzip.GzipFile(fileobj=StringIO(data.read()))
+ gfile = gzip.GzipFile(fileobj=streamIO(data.read()))
try:
- return gfile.readlines()
+ if sys.version_info.major == 3:
+ return io.TextIOWrapper(gfile).readlines()
+ else:
+ return gfile.readlines()
finally:
gfile.close()
finally:
@@ -336,7 +384,7 @@ def updateFile(remote, local, verbose=None):
re_whitespace=re.compile('\s+')
try:
- index_url = urllib2.urlopen(index_name, timeout=TIMEOUT)
+ index_url = urlopen(index_name, timeout=TIMEOUT)
index_fields = list(PackageFile(index_name, index_url))
except ParseError:
if verbose:
@@ -419,7 +467,7 @@ def mergeAsSets(*args):
for x in args:
for y in x:
s[y] = True
- l = s.keys()
+ l = list(s.keys())
l.sort()
return l
@@ -572,7 +620,7 @@ def test():
assert readLinesSHA1(['1\n', '23\n']) \
== '14293c9bd646a15dc656eaf8fba95124020dfada'
- file_a = map(lambda x: "%d\n" % x, range(1, 18))
+ file_a = list(map(lambda x: "%d\n" % x, range(1, 18)))
file_b = ['0\n', '1\n', '<2>\n', '<3>\n', '4\n', '5\n', '7\n', '8\n',
'11\n', '12\n', '<13>\n', '14\n', '15\n', 'A\n', 'B\n', 'C\n',
'16\n', '17\n',]
=====================================
lib/python/helpers.py
=====================================
@@ -0,0 +1,7 @@
+# helpers.py -- utility functions that don't belong elsewhere
+
+def isstring(s):
+ try:
+ return isinstance(s, basestring)
+ except NameError:
+ return isinstance(s, str)
=====================================
lib/python/sectracker/xpickle.py
=====================================
@@ -19,7 +19,7 @@ from __future__ import with_statement
import errno as _errno
import os as _os
-import cPickle as _pickle
+import pickle as _pickle
import tempfile as _tempfile
EXTENSION = '.xpck'
=====================================
lib/python/security_db.py
=====================================
@@ -32,15 +32,13 @@ import apsw
import base64
import bugs
from collections import namedtuple
-import cPickle
-import cStringIO
+import pickle
import glob
import itertools
import os
import os.path
import re
import sys
-import types
import zlib
import config
@@ -54,7 +52,7 @@ class InsertError(Exception):
def __init__(self, errors):
assert len(errors) > 0, errors
- assert type(errors) == types.ListType, errors
+ assert isinstance(errors, list), errors
self.errors = errors
def __str__(self):
@@ -62,12 +60,12 @@ class InsertError(Exception):
def mergeLists(a, b):
"""Merges two lists."""
- if type(a) == types.UnicodeType:
+ if isstring(a):
if a == "":
a = []
else:
a = a.split(',')
- if type(b) == types.UnicodeType:
+ if isstring(b):
if b == "":
b = []
else:
@@ -77,7 +75,7 @@ def mergeLists(a, b):
result[x] = 1
for x in b:
result[x] = 1
- result = result.keys()
+ result = list(result.keys())
result.sort()
return result
@@ -554,7 +552,7 @@ class DB:
for arch in arg.split(','):
lst[arch] = True
def string_set_to_archs(lst):
- l = lst.keys()
+ l = list(lst.keys())
l.sort()
return ','.join(l)
def string_set_factory():
@@ -697,15 +695,14 @@ class DB:
return data
def toString(data):
- result = cStringIO.StringIO()
- cPickle.dump(data, result)
- return buffer(result.getvalue())
+ return pickle.dumps(data)
for (old_print, contents) in cursor.execute(
"SELECT inodeprint, parsed FROM inodeprints WHERE file = ?",
(filename,)):
if old_print == current_print:
- return (True, cPickle.load(cStringIO.StringIO(contents)))
+ print(pickle.loads(contents))
+ return (True, pickle.loads(contents))
result = do_parse(debian_support.PackageFile(filename))
cursor.execute("""UPDATE inodeprints SET inodeprint = ?, parsed = ?
WHERE file = ?""", (current_print, toString(result), filename))
@@ -835,7 +832,7 @@ class DB:
cursor.execute("DELETE FROM binary_packages")
self._clearVersions(cursor)
- l = packages.keys()
+ l = list(packages.keys())
if len(l) == 0:
raise ValueError("no binary packages found")
@@ -843,7 +840,7 @@ class DB:
l.sort()
def gen():
for key in l:
- archs = packages[key].keys()
+ archs = list(packages[key].keys())
archs.sort()
archs = ','.join(archs)
yield key + (archs,)
@@ -1361,7 +1358,7 @@ class DB:
# note/release/subrelease triple, but we should check that
# here.
- status = {'' : {}, 'security' : {}, 'lts' : {}}
+ status = {'' : {}, 'security' : {}, 'lts' : {}}
for (package, note, subrelease, vulnerable, urgency) in cursor.execute(
"""SELECT DISTINCT sp.name, n.id, sp.subrelease,
st.vulnerable, n.urgency
@@ -1393,11 +1390,11 @@ class DB:
elif vulnerable == 2:
undet_pkgs[package] = True
- unfixed_pkgs = unfixed_pkgs.keys()
+ unfixed_pkgs = list(unfixed_pkgs.keys())
unfixed_pkgs.sort()
- undet_pkgs = undet_pkgs.keys()
+ undet_pkgs = list(undet_pkgs.keys())
undet_pkgs.sort()
- unimp_pkgs = unimp_pkgs.keys()
+ unimp_pkgs = list(unimp_pkgs.keys())
unimp_pkgs.sort()
pkgs = ""
@@ -1534,7 +1531,7 @@ class DB:
kind, urgency_to_flag[urgency], remote,
fix_available,
package, fixed_version, description))
- result = base64.encodestring(zlib.compress(''.join(result), 9))
+ result = base64.encodestring(zlib.compress(''.join(result).encode('utf-8'), 9))
c.execute(
"INSERT OR REPLACE INTO debsecan_data (name, data) VALUES (?, ?)",
@@ -1689,7 +1686,7 @@ class DB:
elif is_unknown:
bs_flag = ' '
- other_versions = other_versions.keys()
+ other_versions = list(other_versions.keys())
other_versions.sort()
other_versions = ' '.join(other_versions)
@@ -1701,11 +1698,11 @@ class DB:
",%s,%s"
% (unstable_fixed, other_versions)))
fill_vuln_list()
- source_packages = source_packages.keys()
+ source_packages = list(source_packages.keys())
source_packages.sort()
def store_value(name, value):
- value = base64.encodestring(zlib.compress(value, 9))
+ value = base64.encodestring(zlib.compress(value.encode('utf-8'), 9))
c.execute("""INSERT OR REPLACE INTO debsecan_data
VALUES (?, ?)""", (name, value))
@@ -1756,7 +1753,7 @@ class DB:
"""Returns the debsecan data item NAME."""
for (data,) in self.cursor().execute(
"SELECT data FROM debsecan_data WHERE name = ?", (name,)):
- return base64.decodestring(data)
+ return base64.decodestring(data.encode('utf-8'))
else:
return None
=====================================
lib/python/web_support.py
=====================================
@@ -16,7 +16,6 @@
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
import cgi
-import cStringIO
import os
import re
import socket
@@ -24,11 +23,26 @@ import struct
import sys
import grp
import traceback
-import types
-import urllib
import threading
-import SocketServer
-import BaseHTTPServer
+
+try:
+ from urllib import quote as urllib_quote
+except ImportError:
+ from urllib.parse import quote as urllib_quote
+
+try:
+ from SocketServer import ThreadingMixIn
+ from BaseHTTPServer import HTTPServer, BaseHTTPRequestHandler
+except ImportError:
+ from socketserver import ThreadingMixIn
+ from http.server import HTTPServer, BaseHTTPRequestHandler
+
+try:
+ from cStringIO import StringIO
+except ImportError:
+ from io import StringIO
+
+from helpers import isstring
class ServinvokeError(Exception):
pass
@@ -66,7 +80,7 @@ class Service:
gid = os.stat(self.socket_name).st_gid
grpent = grp.getgrgid(gid)
if grpent[0] == 'www-data':
- os.chmod(self.socket_name, 0660)
+ os.chmod(self.socket_name, 0o660)
def log(self, msg, *args):
sys.stderr.write((msg % args) + "\n")
@@ -110,7 +124,7 @@ class Service:
else:
break
data = ''.join(data)
- result = cStringIO.StringIO()
+ result = StringIO()
self.handle(cli, env, data, result)
client.sendall(result.getvalue())
client.close()
@@ -123,7 +137,7 @@ class Service:
raise
except:
client.close()
- target = cStringIO.StringIO()
+ target = StringIO()
traceback.print_exc(None, target)
self.log("%s", target.getvalue())
@@ -164,11 +178,11 @@ class URLFactory:
for (key, value) in args.items():
if value is None:
continue
- if type(value) not in (types.ListType, types.TupleType):
+ if not isinstance(value, (list, tuple)):
value = (value,)
for v in value:
- arglist.append("%s=%s" % (urllib.quote(key),
- urllib.quote(v)))
+ arglist.append("%s=%s" % (urllib_quote(key),
+ urllib_quote(v)))
if arglist:
return "?" + '&'.join(arglist)
else:
@@ -251,7 +265,7 @@ class HTMLBase:
def toString(self):
"""Invokes flatten to create a new string object."""
- r = cStringIO.StringIO()
+ r = StringIO()
self.flatten(r.write)
return r.getvalue()
@@ -268,8 +282,6 @@ class VerbatimHTML(HTMLBase):
def flatten(self, write):
write(self.__contents)
-_string_types = (types.StringType, types.UnicodeType)
-
class Compose(HTMLBase):
"""Glues a sequence of HTML snippets together, without enclosing it in
a tag."""
@@ -278,7 +290,7 @@ class Compose(HTMLBase):
def flatten(self, write):
for x in self.__contents:
- if type(x) in _string_types:
+ if isstring(x):
write(escapeHTML(x))
else:
x.flatten(write)
@@ -325,7 +337,7 @@ class Tag(HTMLBase):
closing = "</%s>" % self.__name
try:
for x in self.contents:
- if type(x) in _string_types:
+ if isstring(x):
write(escapeHTML(x))
else:
x.flatten(write)
@@ -347,7 +359,7 @@ class Tag(HTMLBase):
return "<websupport.Tag instance, name=%s>" % repr(self.__name)
def toString(self):
- r = cStringIO.StringIO()
+ r = StringIO()
self.flatten(r.write)
return r.getvalue()
@@ -456,7 +468,7 @@ def make_table(contents, title=None, caption=None, replacement=None, introductio
cols.append(TD(col))
rows.append(Tag('tr', cols))
if rows:
- if not introduction:
+ if not introduction:
introduction=''
if not title:
title=''
@@ -494,7 +506,7 @@ def make_menu(convert, *entries):
ul = []
append = ul.append
for e in entries:
- if type(e) == types.TupleType:
+ if isinstance(e, tuple):
(relurl, label) = e
append(LI(A(convert(relurl), label)))
else:
@@ -512,14 +524,14 @@ def make_numbered_list(entries):
def make_list(lst, separator=", "):
"""Creates a list of HTML elements."""
- assert type(lst) != types.StringType
+ assert not isstring(lst)
c = []
if lst:
- append = c.append
- for e in lst[:-1]:
- append(e)
- append(separator)
- append(lst[-1])
+ for e in lst:
+ c.append(e)
+ c.append(separator)
+ # pop the final separator
+ c.pop()
return Compose(c)
class InvalidPath(Exception):
@@ -635,6 +647,12 @@ class RedirectResult(Result):
self.status = 302
self.headers['Location'] = str(url)
+def maybe_encode(obj):
+ try:
+ return obj.encode()
+ except:
+ return obj
+
class HTMLResult(Result):
"""An object of this class combines a status code with HTML contents."""
def __init__(self, contents, doctype='', status=200):
@@ -653,16 +671,20 @@ class HTMLResult(Result):
def flatten_later(self):
headers_later = super(HTMLResult, self).flatten_later()
- buf = cStringIO.StringIO()
+ buf = StringIO()
buf.write(self.doctype)
buf.write('\n')
def write_both(s):
- if type(s) == types.UnicodeType:
- buf.write(s.encode('UTF-8'))
- else:
+ try:
+ if isinstance(s, unicode):
+ s = s.encode('UTF-8')
+ except:
+ pass
+ finally:
buf.write(s)
self.contents.flatten(write_both)
buf = buf.getvalue()
+ buf = maybe_encode(buf)
self.headers['Content-Length'] = str(len(buf))
def later(req):
headers_later(req)
@@ -690,7 +712,7 @@ class BinaryResult(Result):
def later(req):
headers_later(req)
if req.command != 'HEAD':
- req.wfile.write(self.contents)
+ req.wfile.write(maybe_encode(self.contents))
return later
class WebServiceBase:
@@ -773,8 +795,7 @@ class WebService(Service, WebServiceBase):
assert isinstance(r, Result), repr(r)
r.flatten(result.write)
-class ThreadingHTTPServer(SocketServer.ThreadingMixIn,
- BaseHTTPServer.HTTPServer):
+class ThreadingHTTPServer(ThreadingMixIn, HTTPServer):
pass
RE_BASE_URL = re.compile(r'^(https?)://([^/]+)(.*)')
@@ -788,7 +809,7 @@ class WebServiceHTTP(WebServiceBase):
self.__parse_base_url(base_url)
service_self = self
- class Handler(BaseHTTPServer.BaseHTTPRequestHandler):
+ class Handler(BaseHTTPRequestHandler):
def do_GET(self):
(method, path, remaining, params) = self.route()
if path is None:
@@ -893,7 +914,7 @@ def __test():
== '<td><a href="http://www.example.net/">example</a></td>'
#assert make_pre(['a', 'b']).toString() == '<pre>a\nb\n</pre>'
- s = cStringIO.StringIO()
+ s = StringIO()
RedirectResult(u.scriptRelativeFull("123")).flatten(s.write)
assert s.getvalue() == '''Location: http://localhost.localdomain/cgi-bin/test.cgi/123
View it on GitLab: https://salsa.debian.org/security-tracker-team/security-tracker/-/compare/3cc5ad8c15a108c676e080a5572d01061a155c25...dd72ac01207a0d7ca14243bb809cd5ee3f7eaec4
--
View it on GitLab: https://salsa.debian.org/security-tracker-team/security-tracker/-/compare/3cc5ad8c15a108c676e080a5572d01061a155c25...dd72ac01207a0d7ca14243bb809cd5ee3f7eaec4
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/debian-security-tracker-commits/attachments/20200729/438d0153/attachment-0001.html>
More information about the debian-security-tracker-commits
mailing list