[Git][security-tracker-team/security-tracker][master] 33 commits: gen-DSA: get distro info from config.json
Salvatore Bonaccorso
carnil at debian.org
Thu Jun 4 21:02:52 BST 2020
Salvatore Bonaccorso pushed to branch master at Debian Security Tracker / security-tracker
Commits:
70c1a8a4 by Emilio Pozuelo Monfort at 2020-02-26T12:31:30+01:00
gen-DSA: get distro info from config.json
- - - - -
47899605 by Emilio Pozuelo Monfort at 2020-02-26T12:31:30+01:00
config.py: add python module to read config.json
- - - - -
75996de1 by Emilio Pozuelo Monfort at 2020-02-26T12:31:30+01:00
tracker_service: don't register oldoldstable when not supported
- - - - -
2a9adcff by Emilio Pozuelo Monfort at 2020-02-26T12:31:30+01:00
tracker_service: don't hardcode backport codenames
- - - - -
b83de2f3 by Emilio Pozuelo Monfort at 2020-02-26T12:31:30+01:00
tracker_service: don't hardcode codenames in db queries
- - - - -
c143d8a9 by Emilio Pozuelo Monfort at 2020-02-26T12:31:30+01:00
security_db: don't hardcode codenames in calls to _calcTesting()
- - - - -
14dba5fc by Emilio Pozuelo Monfort at 2020-02-26T12:31:30+01:00
security_db: don't hardcode codenames in calls to gen_release
- - - - -
27c7e220 by Emilio Pozuelo Monfort at 2020-02-26T12:31:30+01:00
security_db: take the sid value in calculateDebsecan0
When the release is sid, just pass 'sid' rather than the empty
string to change that afterwards.
- - - - -
06a39ee3 by Emilio Pozuelo Monfort at 2020-02-26T12:31:30+01:00
security_db: don't hardcode release codenames in calculateDebsecan
- - - - -
891dbf39 by Emilio Pozuelo Monfort at 2020-02-26T12:31:30+01:00
security_db: don't hardcode releases in db queries
- - - - -
0080683f by Emilio Pozuelo Monfort at 2020-02-26T12:31:30+01:00
security_db: don't hardcode release codenames in _initViews
- - - - -
32c5f4c3 by Emilio Pozuelo Monfort at 2020-02-26T12:31:30+01:00
security_db: remove unused getEffectiveVersion method
- - - - -
4c113abe by Emilio Pozuelo Monfort at 2020-02-26T12:31:30+01:00
dist_config.py: remove unused file
- - - - -
a6857902 by Emilio Pozuelo Monfort at 2020-02-26T12:31:30+01:00
security_db: don't hardcode the testing suite codename
- - - - -
b121f8f7 by Emilio Pozuelo Monfort at 2020-02-26T12:31:30+01:00
security_db: drop squeeze workarounds
- - - - -
70f21121 by Emilio Pozuelo Monfort at 2020-02-26T12:31:30+01:00
config: add a method to get all releases
- - - - -
4812a437 by Emilio Pozuelo Monfort at 2020-02-26T12:31:30+01:00
debian_support: don't hardcode release names
- - - - -
b9e80cfc by Emilio Pozuelo Monfort at 2020-02-26T12:31:30+01:00
security_db: don't hardcode release names
- - - - -
6d4c1779 by Emilio Pozuelo Monfort at 2020-02-26T12:31:30+01:00
tracker_service: unify *stable methods
- - - - -
21813051 by Emilio Pozuelo Monfort at 2020-02-26T12:31:30+01:00
tracker_service: dynamically register stable releases
- - - - -
57efb5f3 by Emilio Pozuelo Monfort at 2020-02-26T12:31:30+01:00
tracker_service: simplify stable-like callbacks
And take the file out of README.releases.
- - - - -
ad210ce2 by Emilio Pozuelo Monfort at 2020-02-26T12:31:30+01:00
Makefile: remove update-$(alias) targets
They are aliases for the real update-$(dist) targets, and
are unused as we mostly use update-packages target, which
directly calls the update-$(dist) ones.
- - - - -
0d22e6b8 by Emilio Pozuelo Monfort at 2020-02-26T12:31:30+01:00
Don't hardcode architecture list in the Makefile
Move it to config.json instead and grab it from there.
- - - - -
23baa131 by Emilio Pozuelo Monfort at 2020-02-26T12:31:30+01:00
Makefile: don't hardcode Debian releases
- - - - -
899dbea2 by Emilio Pozuelo Monfort at 2020-02-26T12:31:30+01:00
README.releases: update config.json on new releases
- - - - -
cffd0233 by Emilio Pozuelo Monfort at 2020-02-26T12:31:30+01:00
lts-needs-forward-port: take releases from config.py
- - - - -
8ab31264 by Emilio Pozuelo Monfort at 2020-02-26T12:31:30+01:00
lts-needs-forward-port: add FIXME about looking at stable pu list
- - - - -
bc067aea by Emilio Pozuelo Monfort at 2020-02-26T12:31:30+01:00
lts-bts: get LTS release from config.py
- - - - -
7850fe4f by Emilio Pozuelo Monfort at 2020-02-26T12:31:30+01:00
lts-cve-triage: take lts releases from config.py
- - - - -
621333b9 by Emilio Pozuelo Monfort at 2020-02-26T12:31:30+01:00
Call TrackerData's Issue::get_status() with release codenames
- - - - -
129cdcc5 by Emilio Pozuelo Monfort at 2020-02-26T12:31:30+01:00
tracker_data: remove RELEASES array
This also removes the normalize_release method, and Issue::get_status()
no longer supports passing aliases such as 'stable' or 'lts'.
- - - - -
1b9c4741 by Emilio Pozuelo Monfort at 2020-02-26T12:31:30+01:00
security_db: don't hardcode the list of supported releases
At times there will just be two, so get that list from the
config.
- - - - -
12052760 by Salvatore Bonaccorso at 2020-06-04T20:02:34+00:00
Merge branch 'distro-config' into 'master'
Distro config reunification
See merge request security-tracker-team/security-tracker!48
- - - - -
14 changed files:
- Makefile
- bin/gen-DSA
- bin/lts-bts
- bin/lts-cve-triage.py
- bin/lts-needs-forward-port.py
- bin/tracker_data.py
- bin/tracker_service.py
- data/config.json
- doc/README.releases
- lib/debian-releases.mk
- + lib/python/config.py
- lib/python/debian_support.py
- − lib/python/dist_config.py
- lib/python/security_db.py
Changes:
=====================================
Makefile
=====================================
@@ -1,23 +1,8 @@
PYTHON_MODULES = $(wildcard lib/python/*.py)
-# The following variables need to be kept up-to-date and can be adjusted
-# currently unsupported releases can be commented out
-OLDOLDSTABLE = jessie
-OLDSTABLE = stretch
-STABLE = buster
-TESTING = bullseye
-
MIRROR = http://debian.csail.mit.edu/debian
SECURITY_MIRROR = http://security.debian.org/debian-security
-jessie_ARCHS = amd64 armel armhf i386
-stretch_ARCHS = amd64 arm64 armel armhf i386 mips mips64el mipsel ppc64el s390x
-buster_ARCHS = amd64 arm64 armel armhf i386 mips mips64el mipsel ppc64el s390x
-bullseye_ARCHS = amd64 arm64 armel armhf i386 mips64el mipsel ppc64el s390x
-sid_ARCHS = amd64 arm64 armel armhf i386 mips64el mipsel ppc64el s390x
-
-# The rest of the file should not need to be edited
-
# Include the definitions of the releases to be fetched
include lib/*-releases.mk
@@ -78,30 +63,12 @@ endef
$(foreach release,$(RELEASES),$(eval $(call add_update_rule,$(release))))
# Define some common aliases
-.PHONY: update-unstable update-testing update-stable update-oldstable update-oldoldstable
-.PHONY: update-testing-security update-stable-security update-oldstable-security update-oldoldstable-security
.PHONY: update-main update-security update-backports
-update-unstable: update-sid
-update-testing: update-$(TESTING)
-update-testing-security: update-$(TESTING)_security
-update-stable: update-$(STABLE)
-update-stable-security: update-$(STABLE)_security
-update-oldstable: update-$(OLDSTABLE)
-update-oldstable-security: update-$(OLDSTABLE)_security
-ifeq ($(OLDOLDSTABLE),)
-update-oldoldstable:
-update-oldoldstable-security:
-else
-update-oldoldstable: update-$(OLDOLDSTABLE)
-update-oldoldstable-security: update-$(OLDOLDSTABLE)_security
-endif
update-main: $(foreach release,$(MAIN_RELEASES),update-$(release))
update-security: $(foreach release,$(SECURITY_RELEASES),update-$(release)_security)
update-backports: $(foreach release,$(BACKPORT_RELEASES),update-$(release)_backports)
supported-update-targets:
- @echo -n "unstable testing stable oldstable oldoldstable "
- @echo -n "testing-security stable-security oldstable-security oldoldstable-security "
@echo -n "main security backports "
@echo -n "$(RELEASES) "
@echo -n "packages lists nvd"
=====================================
bin/gen-DSA
=====================================
@@ -27,10 +27,20 @@ case "$(basename "$0")" in
;;
esac
-OLDOLDSTABLE=jessie
-OLDSTABLE=stretch
-STABLE=buster
-TESTING=bullseye
+if ! which jq >/dev/null 2>&1 ; then
+ echo "error: jq is needed to parse distributions, please install it"
+ exit 1
+fi
+
+RELEASES=`jq -r '.distributions | to_entries[] | select(.value.release) | .value.release | ascii_upcase' data/config.json`
+CODENAMES=`jq -r '.distributions | to_entries[] | select(.value.release) | .key' data/config.json`
+
+while read dist; do
+ read codename
+ eval $dist=$codename
+done << EOF
+`jq -r '.distributions | to_entries[] | select(.value.release) | (.value.release | ascii_upcase), .key' data/config.json`
+EOF
NAME_SPACING=24
DATE_SPACING=22
@@ -335,15 +345,15 @@ setvar PACKAGE
setvar CVE "$CVE_LIST"
setvar ${IDMODE}ID "$DAID"
setvar BUGNUM
-setvar OLDOLDSTABLE
-setvar OLDSTABLE
-setvar STABLE
-setvar TESTING
setvar SPACEDDATE
setvar DATE
setvar TEXT "${TEXT:-$IDMODE text goes here}"
-for dist in $OLDOLDSTABLE $OLDSTABLE $STABLE $TESTING UNSTABLE; do
+for dist in $RELEASES; do
+ setvar $dist
+done
+
+for dist in $CODENAMES; do
version="$(eval 'printf "%s" "$'"$dist"_VERSION'"')"
if $save && [ -z "$version" ] && grep -q "${dist}_VERSION" "$tmpf"; then
printf "Enter $dist's version [unset]: "
@@ -377,7 +387,7 @@ EOF
printf "\t{%s}\n" "$CVE" >> $daid_entry
fi
- for dist in $OLDOLDSTABLE $OLDSTABLE $STABLE; do
+ for dist in $CODENAMES; do
version="$(eval 'printf "%s" "$'"$dist"_VERSION'"')"
[ -z "$version" ] || \
printf "\t[%s] - %s %s\n" "$dist" "$PACKAGE" "$version" >> $daid_entry
=====================================
bin/lts-bts
=====================================
@@ -11,7 +11,15 @@ import sys
import tempfile
import warnings
-from tracker_data import TrackerData, RELEASES
+from tracker_data import TrackerData
+
+def setup_path():
+ dirname = os.path.dirname
+ base = dirname(dirname(os.path.realpath(sys.argv[0])))
+ sys.path.insert(0, os.path.join(base, "lib", "python"))
+
+setup_path()
+import config
from jinja2 import Template
@@ -103,7 +111,7 @@ def main():
cc = 'debian-lts at lists.debian.org'
team = 'lts'
- release = RELEASES['lts']
+ release = config.get_supported_releases()[0]
# Basic check
instructions = "packages/{}.txt".format(args.package)
=====================================
bin/lts-cve-triage.py
=====================================
@@ -15,13 +15,26 @@
# You should have received a copy of the GNU General Public License
# along with this file. If not, see <https://www.gnu.org/licenses/>.
+import os
import sys
import argparse
import collections
-from tracker_data import TrackerData, RELEASES
+from tracker_data import TrackerData
from unsupported_packages import UnsupportedPackages, LimitedSupportPackages
+def setup_path():
+ dirname = os.path.dirname
+ base = dirname(dirname(os.path.realpath(sys.argv[0])))
+ sys.path.insert(0, os.path.join(base, "lib", "python"))
+
+setup_path()
+import config
+
+RELEASES = {
+ 'lts': config.get_supported_releases()[0],
+ 'next_lts': config.get_supported_releases()[1],
+}
def colored(x, *args, **kwargs):
return x
@@ -100,8 +113,8 @@ for pkg in tracker.iterate_packages():
continue
for issue in tracker.iterate_pkg_issues(pkg):
- status_in_lts = issue.get_status('lts')
- status_in_next_lts = issue.get_status('next_lts')
+ status_in_lts = issue.get_status([RELEASES['lts'])
+ status_in_next_lts = issue.get_status(RELEASES['next_lts'])
if status_in_lts.status in ('not-affected', 'resolved'):
continue
=====================================
bin/lts-needs-forward-port.py
=====================================
@@ -18,21 +18,33 @@
import argparse
import collections
+import os
import sys
-from tracker_data import TrackerData, RELEASES
+from tracker_data import TrackerData
+
+def setup_path():
+ dirname = os.path.dirname
+ base = dirname(dirname(os.path.realpath(sys.argv[0])))
+ sys.path.insert(0, os.path.join(base, "lib", "python"))
+
+setup_path()
+import config
+
+lts = config.get_supported_releases()[0]
+next_lts = config.get_supported_releases()[1]
+oldstable = config.get_release_codename('oldstable')
-# lts is currently jessie, next_lts stretch
LIST_NAMES = (
('needs_fix_in_next_lts',
- ('Issues that are unfixed in {next_lts} but fixed in {lts}'
- ).format(**RELEASES)),
+ ('Issues that are unfixed in {} but fixed in {}'
+ ).format(next_lts, lts)),
('needs_review_in_next_lts',
- ('Issues that are no-dsa in {next_lts} but fixed in {lts}'
- ).format(**RELEASES)),
+ ('Issues that are no-dsa in {} but fixed in {}'
+ ).format(next_lts, lts)),
('fixed_via_pu_in_oldstable',
- ('Issues that will be fixed via p-u in {oldstable}'
- ).format(**RELEASES)),
+ ('Issues that will be fixed via p-u in {}'
+ ).format(oldstable)),
)
@@ -55,8 +67,8 @@ def main():
for pkg in tracker.iterate_packages():
for issue in tracker.iterate_pkg_issues(pkg):
- status_in_lts = issue.get_status('lts')
- status_in_next_lts = issue.get_status('next_lts')
+ status_in_lts = issue.get_status(lts)
+ status_in_next_lts = issue.get_status(next_lts)
if status_in_lts.status in ('not-affected', 'open'):
continue
@@ -64,6 +76,7 @@ def main():
if status_in_lts.status == 'resolved':
# Package will be updated via the next oldstable
# point release
+ # FIXME: when lts == oldstable, this should look at the stable pu list
if (issue.name in tracker.oldstable_point_update and
pkg in tracker.oldstable_point_update[issue.name]):
add_to_list('fixed_via_pu_in_oldstable', pkg, issue)
=====================================
bin/tracker_data.py
=====================================
@@ -21,27 +21,6 @@ import subprocess
import requests
import six
-RELEASES = {
- 'oldoldstable': 'jessie',
- 'oldstable': 'stretch',
- 'stable': 'buster',
- 'testing': 'bullseye',
- 'unstable': 'sid',
- 'experimental': 'experimental',
- # LTS specific aliases
- 'lts': 'jessie',
- 'next_lts': 'stretch',
-}
-
-
-def normalize_release(release):
- if release in RELEASES:
- return RELEASES[release]
- elif release in RELEASES.values():
- return release
- else:
- raise ValueError("Unknown release: {}".format(release))
-
class TrackerData(object):
DATA_URL = "https://security-tracker.debian.org/tracker/data/json"
@@ -189,7 +168,6 @@ class Issue(object):
self.data = data
def get_status(self, release):
- release = normalize_release(release)
data = self.data['releases'].get(release)
if data is None:
status = 'not-affected'
=====================================
bin/tracker_service.py
=====================================
@@ -3,6 +3,7 @@
import sys
sys.path.insert(0,'../lib/python')
import bugs
+import config
import re
import security_db
from web_support import *
@@ -138,21 +139,24 @@ class TrackerService(webservice_base_class):
self.json_data = None # the JSON dump itself
self.json_timestamp = None # timestamp of JSON generation
self.json_last_modified = None
+
+ self.stable_releases = config.get_supported_releases()
+ self.stable_releases.remove(config.get_release_codename('testing'))
+ self.stable_releases.remove('sid')
+ self.stable_releases.reverse()
+
self.register('', self.page_home)
self.register('*', self.page_object)
self.register('redirect/*', self.page_redirect)
self.register('source-package/*', self.page_source_package)
- self.register('status/release/oldoldstable',
- self.page_status_release_oldoldstable)
- self.register('status/release/oldstable',
- self.page_status_release_oldstable)
- self.register('status/release/stable', self.page_status_release_stable)
- self.register('status/release/stable-backports',
- self.page_status_release_stable_backports)
- self.register('status/release/oldstable-backports',
- self.page_status_release_oldstable_backports)
- self.register('status/release/oldoldstable-backports',
- self.page_status_release_oldoldstable_backports)
+
+ for release in self.stable_releases:
+ alias = config.get_release_alias(release)
+ self.register('status/release/' + alias,
+ self.page_status_release_stable_like)
+ self.register('status/release/' + alias + '-backports',
+ self.page_status_release_backports_like)
+
self.register('status/release/testing',
self.page_status_release_testing)
self.register('status/release/unstable',
@@ -213,6 +217,16 @@ class TrackerService(webservice_base_class):
else:
return RedirectResult(url.scriptRelativeFull(query))
+ def gen_stable_links():
+ links = []
+ for release in self.stable_releases:
+ alias = config.get_release_alias(release)
+ links.append(('status/release/' + alias,
+ 'Vulnerable packages in the ' + alias + ' suite'))
+ links.append(('status/release/' + alias + '-backports',
+ 'Vulnerable packages in backports for ' + alias))
+ return links
+
return self.create_page(
url, 'Security Bug Tracker',
[P(
@@ -238,23 +252,12 @@ aware of and/or help us improve the quality of this information by """,
NAV(make_menu(
url.scriptRelative,
- ('status/release/unstable',
+ *[('status/release/unstable',
'Vulnerable packages in the unstable suite'),
('status/release/testing',
- 'Vulnerable packages in the testing suite'),
- ('status/release/stable',
- 'Vulnerable packages in the stable suite'),
- ('status/release/stable-backports',
- 'Vulnerable packages in backports for stable'),
- ('status/release/oldstable',
- 'Vulnerable packages in the oldstable suite'),
- ('status/release/oldstable-backports',
- 'Vulnerable packages in backports for oldstable'),
- ('status/release/oldoldstable',
- 'Vulnerable packages in the oldoldstable suite'),
- ('status/release/oldoldstable-backports',
- 'Vulnerable packages in backports for oldoldstable'),
- ('status/dtsa-candidates', "Candidates for DTSAs"),
+ 'Vulnerable packages in the testing suite')]
+ + gen_stable_links() +
+ [('status/dtsa-candidates', "Candidates for DTSAs"),
('status/todo', 'TODO items'),
('status/undetermined', 'Packages that may be vulnerable but need to be checked (undetermined issues)'),
('status/unimportant', 'Packages that have open unimportant issues'),
@@ -273,7 +276,7 @@ aware of and/or help us improve the quality of this information by """,
'Covered Debian releases and architectures'),
('data/json',
'All information in JSON format')
- )),
+ ])),
self.make_search_button(url),
P("""(You can enter CVE names, Debian bug numbers and package
@@ -693,8 +696,8 @@ to improve our documentation and procedures, so feedback is welcome.""")])])
replacement='No known security announcements.')
])
- def page_status_release_stable_oldstable_oldoldstable(self, release, params, url):
- assert release in ('stable', 'oldstable', 'oldoldstable',)
+ def page_status_release_stable_like(self, path, params, url):
+ release = os.path.basename(url.path_info)
bf = BugFilter(params)
@@ -750,15 +753,6 @@ to improve our documentation and procedures, so feedback is welcome.""")])])
for this vulnerability.'''),
self.nvd_text])
- def page_status_release_stable(self, path, params, url):
- return self.page_status_release_stable_oldstable_oldoldstable('stable', params, url)
- def page_status_release_oldstable(self, path, params, url):
- return self.page_status_release_stable_oldstable_oldoldstable('oldstable',
- params, url)
- def page_status_release_oldoldstable(self, path, params, url):
- return self.page_status_release_stable_oldstable_oldoldstable('oldoldstable',
- params, url)
-
def page_status_release_testing(self, path, params, url):
bf = BugFilter(params)
@@ -878,24 +872,14 @@ to improve our documentation and procedures, so feedback is welcome.""")])])
title='Vulnerable source packages in the unstable suite',
rel='sid')
- def page_status_release_stable_backports(self, path, params, url):
- return self.page_status_release_unstable_like(
- path, params, url,
- title='Vulnerable source packages among backports for stable',
- rel='buster-backports')
-
- def page_status_release_oldstable_backports(self, path, params, url):
- return self.page_status_release_unstable_like(
- path, params, url,
- title='Vulnerable source packages among backports for oldstable',
- rel='stretch-backports')
+ def page_status_release_backports_like(self, path, params, url):
+ release = os.path.basename(url.path_info)
+ release = release.split("-")[0]
- def page_status_release_oldoldstable_backports(self, path, params, url):
return self.page_status_release_unstable_like(
path, params, url,
- title='Vulnerable source packages among backports for oldoldstable',
- rel='jessie-backports')
-
+ title='Vulnerable source packages among backports for ' + release,
+ rel=config.get_release_codename(release, '-backports'))
def page_status_dtsa_candidates(self, path, params, url):
bf = BugFilter(params,nonodsa=True,noignored=True,nopostponed=True)
@@ -909,18 +893,19 @@ to improve our documentation and procedures, so feedback is welcome.""")])])
(SELECT testing.version_id < stable.version_id
FROM source_packages AS testing, source_packages AS stable
WHERE testing.name = testing_status.package
- AND testing.release = 'bullseye'
+ AND testing.release = ?
AND testing.subrelease = ''
AND testing.archive = testing_status.section
AND stable.name = testing_status.package
- AND stable.release = 'buster'
+ AND stable.release = ?
AND stable.subrelease = 'security'
AND stable.archive = testing_status.section),
(SELECT range_remote FROM nvd_data
WHERE cve_name = bug)
FROM testing_status
WHERE (NOT unstable_vulnerable)
- AND (NOT testing_security_fixed)"""):
+ AND (NOT testing_security_fixed)""",
+ (config.get_release_codename('testing'), config.get_release_codename('stable'))):
if bf.urgencyFiltered(urgency, vulnerable):
continue
if bf.remoteFiltered(remote):
@@ -994,14 +979,13 @@ checker to find out why they have not entered testing yet."""),
old_pkg = ''
old_dsc = ''
last_displayed = ''
- releases = ('sid', 'bullseye', 'buster', 'stretch', 'jessie')
+ releases = config.get_supported_releases()
for (pkg_name, bug_name, release, desc) in self.db.cursor().execute(
"""SELECT DISTINCT sp.name, st.bug_name, sp.release,
bugs.description
FROM source_package_status AS st, source_packages AS sp, bugs
WHERE st.vulnerable == 2 AND sp.rowid = st.package
- AND ( sp.release = ? OR sp.release = ? OR sp.release = ?
- OR sp.release = ? OR sp.release = ? )
+ AND sp.release IN (""" + ",".join("?" * len(releases)) + """)
AND sp.subrelease = '' AND st.bug_name == bugs.name
ORDER BY sp.name, st.bug_name""", releases):
@@ -1039,14 +1023,14 @@ checker to find out why they have not entered testing yet."""),
old_dsc = ''
old_name = ''
last_displayed = ''
- releases = ('sid', 'bullseye', 'buster', 'stretch', 'jessie')
+ releases = config.get_supported_releases()
for (pkg_name, bug_name, release, desc) in self.db.cursor().execute(
"""SELECT DISTINCT sp.name, st.bug_name, sp.release,
bugs.description
FROM source_package_status AS st, source_packages AS sp, bugs
WHERE st.vulnerable > 0 AND sp.rowid = st.package
- AND ( sp.release = ? OR sp.release = ? OR sp.release = ?
- OR sp.release = ? OR sp.release = ? ) AND st.urgency == 'unimportant'
+ AND sp.release IN (""" + ",".join("?" * len(releases)) + """)
+ AND st.urgency == 'unimportant'
AND sp.subrelease = '' AND st.bug_name == bugs.name
ORDER BY sp.name, st.bug_name""", releases):
@@ -1325,7 +1309,7 @@ Debian bug number.'''),
urgency = defaultdict(lambda: defaultdict(dict))
nodsa = defaultdict(lambda: defaultdict(dict))
nodsa_reason = defaultdict(lambda: defaultdict(dict))
- supported_releases = ('sid', 'bullseye', 'buster', 'stretch', 'jessie')
+ supported_releases = config.get_supported_releases()
for (pkg, issue, desc, debianbug, release, subrelease, db_version, db_fixed_version, db_status, db_urgency, db_remote, db_nodsa, db_nodsa_reason) in self.db.cursor().execute(
"""SELECT sp.name, st.bug_name,
(SELECT cve_desc FROM nvd_data
@@ -1350,8 +1334,7 @@ Debian bug number.'''),
FROM source_package_status AS st, source_packages AS sp, bugs
WHERE sp.rowid = st.package AND st.bug_name = bugs.name
AND ( st.bug_name LIKE 'CVE-%' OR st.bug_name LIKE 'TEMP-%' )
- AND ( sp.release = ? OR sp.release = ? OR sp.release = ?
- OR sp.release = ? OR sp.release = ? )
+ AND sp.release IN (""" + ",".join("?" * len(supported_releases)) + """)
ORDER BY sp.name, st.bug_name, sp.release, sp.subrelease""" , supported_releases):
### to ease debugging...:
=====================================
data/config.json
=====================================
@@ -59,6 +59,7 @@
"jessie-proposed-updates"
]
},
+ "architectures": [ "amd64", "armel", "armhf", "i386" ],
"release": "oldoldstable"
},
"stretch": {
@@ -71,6 +72,7 @@
"stretch-proposed-updates"
]
},
+ "architectures": [ "amd64", "arm64", "armel", "armhf", "i386", "mips", "mips64el", "mipsel", "ppc64el", "s390x" ],
"release": "oldstable"
},
"buster": {
@@ -83,6 +85,7 @@
"buster-proposed-updates"
]
},
+ "architectures": [ "amd64", "arm64", "armel", "armhf", "i386", "mips", "mips64el", "mipsel", "ppc64el", "s390x" ],
"release": "stable"
},
"bullseye": {
@@ -95,6 +98,7 @@
"bullseye-proposed-updates"
]
},
+ "architectures": [ "amd64", "arm64", "armel", "armhf", "i386", "mips64el", "mipsel", "ppc64el", "s390x" ],
"release": "testing"
},
"bookworm": {
@@ -114,6 +118,7 @@
"sid"
]
},
+ "architectures": [ "amd64", "arm64", "armel", "armhf", "i386", "mips64el", "mipsel", "ppc64el", "s390x" ],
"release": "unstable"
}
},
=====================================
doc/README.releases
=====================================
@@ -1,24 +1,16 @@
Checklist to perform when a new stable release is announced
===========================================================
+See https://bugs.debian.org/783491
+
General
-------
[ ] Update doc/DSA.template
-[ ] Update bin/gen-DSA
[ ] bin/add-dsa-needed.sh
-[ ] bin/tracker_data.py
+[ ] data/config.json
[ ] Update security-team.debian.org pages
[ ] Update support information in static/distributions.json
-Security Tracker code
----------------------
-See https://bugs.debian.org/783491
-[ ] bin/tracker_service.py
-[ ] lib/python/debian_support.py
-[ ] lib/python/dist_config.py
-[ ] lib/python/security_db.py
-[ ] Makefile
-
Security Tracker host
---------------------
[ ] Check /srv/security-tracker.debian.org/website/bin
=====================================
lib/debian-releases.mk
=====================================
@@ -1,18 +1,22 @@
# This file defines the variables describing all Debian repositories
# that need to be fetched in the "update-packages" process
+define get_config =
+$(shell jq -r $(1) 'data/config.json')
+endef
+
# backports suites only have Sources.xz and respective Packages.xz
# available.
# Cf. as well https://bugs.debian.org/664866
#BACKPORT_RELEASES := $(OLDSTABLE) $(STABLE)
-SECURITY_RELEASES := $(OLDOLDSTABLE) $(OLDSTABLE) $(STABLE) $(TESTING)
-MAIN_RELEASES := $(SECURITY_RELEASES) sid
+MAIN_RELEASES = $(call get_config, '.distributions | to_entries[] | select(.value.release) | .key')
+SECURITY_RELEASES = $(filter-out sid, $(MAIN_RELEASES))
# Define the variables for the release on the main mirror
define add_main_release =
$(1)_MIRROR = $$(MIRROR)
$(1)_DIST = $(1)
-$(1)_ARCHS ?= amd64 arm64 armel armhf i386 mips64el mipsel ppc64el s390x
+$(1)_ARCHS = $(call get_config, '.distributions.$(1).architectures[]')
$(1)_RELEASE = $(1)
$(1)_SUBRELEASE =
RELEASES += $(1)
=====================================
lib/python/config.py
=====================================
@@ -0,0 +1,59 @@
+# config.py -- methods to read global configuration from data/config.json
+# Copyright (C) 2019 Emilio Pozuelo Monfort <pochu at debian.org>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+# TODO: the OrderedDict use can be dropped once we use Python 3 (>= 3.7)
+from collections import OrderedDict
+import json
+import os
+
+_config = None
+
+def get_config():
+ global _config
+ if not _config:
+ d = os.path.dirname(os.path.abspath(__file__))
+
+ with open(d + '/../../data/config.json') as f:
+ config = json.load(f, object_pairs_hook=OrderedDict)
+
+ _config = config['distributions']
+
+ return _config
+
+def get_supported_releases():
+ config = get_config()
+
+ return [d for d in config.keys() if 'release' in config[d]]
+
+def get_all_releases():
+ config = get_config()
+
+ return config.keys()
+
+def get_release_codename(release, suffix=''):
+ config = get_config()
+
+ for r in config.keys():
+ if 'release' in config[r] and config[r]['release'] == release:
+ return r + suffix
+
+ return None
+
+def get_release_alias(codename):
+ config = get_config()
+
+ return config[codename]['release']
=====================================
lib/python/debian_support.py
=====================================
@@ -37,6 +37,8 @@ except ImportError:
import apt_pkg
apt_pkg.init()
+import config
+
# Timeout for downloads.
TIMEOUT = 30
@@ -194,8 +196,7 @@ class Release(PseudoEnum): pass
def listReleases():
releases = {}
- rels = ("experimental", # For use in [brackets] in the list files.
- "potato", "woody", "sarge", "etch", "lenny", "squeeze", "wheezy", "jessie", "stretch", "buster", "bullseye", "sid")
+ rels = ["experimental"] + config.get_all_releases()
for r in range(len(rels)):
releases[rels[r]] = Release(rels[r], r)
Release.releases = releases
=====================================
lib/python/dist_config.py deleted
=====================================
@@ -1,97 +0,0 @@
-# dist_config.py -- describe how the Debian package database is assembled
-# Copyright (C) 2008 Florian Weimer <fw at deneb.enyo.de>
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program; if not, write to the Free Software
-# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
-
-"""
-This Python moule describes how different views of the Debian package
-database are assembled from a set of on-disk files.
-
-Each view is labeled by a purpose. Currently defined purposes are:
-
- overview: Used to generate the release overview web page. This
- should not contain vulnerabilities which the security team
- considers processed.
-
- debsecan: Used to generate the "fix is available" data for debsecan.
- This should reflect the recommended set of sources.list
- entries for the release.
-"""
-
-######################################################################
-# Configuration section
-######################################################################
-
-def apply_config():
- # Invoked at the end of the file. Edit this to suit your needs.
-
- common_archs = 'amd64,armel,i386,mips,mipsel,powerpc'.split(',')
- squeeze_archs = common_archs + ['s390','ia64','kfreebsd-amd64','kfreebsd-i386','sparc' ]
- wheezy_archs = [ 'amd64','armel','armhf','i386' ]
- jessie_archs = [ 'amd64','armel','armhf','i386' ]
- stretch_archs = [ 'amd64','arm64','armel','armhf','i386','mips','mips64el','mipsel','ppc64el','s390x' ]
- buster_archs = [ 'amd64','arm64','armel','armhf','i386','mips','mips64el','mipsel','ppc64el','s390x' ]
- bullseye_archs = [ 'amd64','arm64','armel','armhf','i386','mips64el','mipsel','ppc64el','s390x' ]
- sid_archs = [ 'amd64','arm64','armel','armhf','i386','mips64el','mipsel','ppc64el','s390x' ]
-
- add_release(name='squeeze',
- architectures=squeeze_archs,
- )
-
- add_release(name='wheezy',
- architectures=wheezy_archs,
- )
-
- add_release(name='jessie',
- architectures=jessie_archs,
- )
-
- add_release(name='stretch',
- architectures=stretch_archs,
- )
-
- add_release(name='buster',
- architectures=buster_archs,
- )
-
- add_release(name='bullseye',
- architectures=bullseye_archs,
- )
-
- add_release(name='sid',
- architectures=sid_archs,
- )
-
-######################################################################
-# Support routines
-######################################################################
-
-releases = {}
-
-def add_release(name, architectures,
- debsecan_part=('', 'security'),
- overview_part=('', 'security', 'proposed-updates')):
- import debian_support
- name = debian_support.internRelease(name)
- if name in releases:
- raise ValueError("duplicate release", name)
- releases[name] = {'architectures' : architectures,
- 'purpose' : {'debsecan' : debsecan_part,
- 'overview' : overview_part}}
-
-# Run the code in the configuration section
-
-apply_config()
-del apply_config
=====================================
lib/python/security_db.py
=====================================
@@ -43,8 +43,8 @@ import sys
import types
import zlib
+import config
import debian_support
-import dist_config
class InsertError(Exception):
"""Class for capturing insert errors.
@@ -464,6 +464,7 @@ class DB:
""")
def _initViews(self, cursor):
+ testing = config.get_release_codename('testing')
cursor.execute(
"""CREATE TEMPORARY VIEW testing_status AS
SELECT DISTINCT sp.name AS package, st.bug_name AS bug,
@@ -479,7 +480,7 @@ class DB:
COALESCE((SELECT NOT vulnerable
FROM source_packages AS tsecp, source_package_status AS tsecst
WHERE tsecp.name = sp.name
- AND tsecp.release = 'bullseye' AND tsecp.subrelease = 'security'
+ AND tsecp.release = '%s' AND tsecp.subrelease = 'security'
AND tsecp.archive = sp.archive
AND tsecst.bug_name = st.bug_name
AND tsecst.package = tsecp.rowid), 0) AS testing_security_fixed,
@@ -488,13 +489,19 @@ class DB:
(EXISTS (SELECT * FROM package_notes_nodsa AS pnd
WHERE pnd.bug_name = st.bug_name
AND pnd.package = sp.name
- AND pnd.release = 'bullseye')) AS no_dsa
+ AND pnd.release = '%s')) AS no_dsa
FROM source_package_status AS st, source_packages AS sp
WHERE st.vulnerable > 0 AND sp.rowid = st.package
- AND sp.release = 'bullseye' AND sp.subrelease = ''
- ORDER BY sp.name, st.urgency, st.bug_name""")
+ AND sp.release = '%s' AND sp.subrelease = ''
+ ORDER BY sp.name, st.urgency, st.bug_name"""
+ % (testing, testing, testing))
- for (name, nickname) in (('stable', 'buster'), ('oldstable', 'stretch'), ('oldoldstable', 'jessie'),):
+ releases = config.get_supported_releases()
+ releases.remove(config.get_release_codename('testing'))
+ releases.remove('sid')
+
+ for release in releases:
+ alias = config.get_release_alias(release)
cursor.execute(
"""CREATE TEMPORARY VIEW %s_status AS
SELECT DISTINCT sp.name AS package, st.bug_name AS bug,
@@ -521,7 +528,7 @@ class DB:
AND secst.bug_name = st.bug_name
AND secst.package = secp.rowid), 0)
ORDER BY sp.name, urgency_to_number(urgency), st.bug_name"""
- % (name, nickname, nickname, nickname, nickname))
+ % (alias, release, release, release, release))
cursor.execute(
"""CREATE TEMPORARY VIEW debian_cve AS
@@ -582,7 +589,7 @@ class DB:
return -1
self.db.createscalarfunction("subreleasepart_to_number", subreleasepart_to_number, 1)
- releases = ['potato', 'woody', 'sarge', 'etch', 'lenny', 'squeeze', 'wheezy', 'jessie', 'stretch', 'buster', 'bullseye', 'sid']
+ releases = config.get_all_releases()
def release_to_number(u):
try:
return releases.index(u)
@@ -740,9 +747,6 @@ class DB:
if unchanged:
continue
- if release == 'squeeze-lts':
- release = 'squeeze'
- subrelease = 'lts'
cursor.execute(
"""DELETE FROM source_packages
WHERE release = ? AND subrelease = ? AND archive = ?""",
@@ -803,9 +807,6 @@ class DB:
raise ValueError("invalid file name: " + repr(filename))
(release, subrelease, archive, architecture) = match.groups()
- if release == 'squeeze-lts':
- release = 'squeeze'
- subrelease = 'lts'
(unch, parsed) = self._parseFile(cursor, filename)
unchanged = unchanged and unch
for name in parsed.keys():
@@ -1140,7 +1141,7 @@ class DB:
"""Calculate vulnerable packages.
To each package note, a release-specific vulnerability status
- is attached. Currently, only bullseye/testing is processed.
+ is attached. Currently, only testing is processed.
Returns a list strings describing inconsistencies.
"""
@@ -1156,17 +1157,18 @@ class DB:
# The following does not work because stable->security ->
# testing -> unstable propagation is no longer available.
if False:
- # Ignore bullseye/testing because stable issues may be
+ # Ignore testing because stable issues may be
# fast-tracked into testing, bypassing unstable.
+ testing = config.get_release_codename('testing')
for (bug_name, pkg_name, rel, unstable_ver, rel_ver) \
in list(cursor.execute(
"""SELECT a.bug_name, a.package, b.release,
a.fixed_version, b.fixed_version
FROM package_notes a, package_notes b
WHERE a.bug_name = b.bug_name AND a.package = b.package
- AND a.release = '' AND b.release NOT IN ('', 'bullseye')
+ AND a.release = '' AND b.release NOT IN ('', '%s')
AND a.fixed_version IS NOT NULL
- AND a.fixed_version_id < b.fixed_version_id""")):
+ AND a.fixed_version_id < b.fixed_version_id""" % (testing,))):
b = bugs.BugFromDB(cursor, bug_name)
result.append("%s:%d: inconsistent versions for package %s"
% (b.source_file, b.source_line, pkg_name))
@@ -1280,10 +1282,13 @@ class DB:
"SELECT name FROM bugs WHERE NOT not_for_us"):
self._calcUnstable(c, bug_name)
- self._calcTesting(c, bug_name, 'testing', 'bullseye')
- self._calcTesting(c, bug_name, 'stable', 'buster')
- self._calcTesting(c, bug_name, 'oldstable', 'stretch')
- self._calcTesting(c, bug_name, 'oldoldstable', 'jessie')
+
+ for release in config.get_supported_releases():
+ if release == 'sid':
+ continue
+
+ alias = config.get_release_alias(release)
+ self._calcTesting(c, bug_name, alias, release)
return result
@@ -1452,12 +1457,10 @@ class DB:
c.execute("""INSERT INTO vulnlist
SELECT bug_name, package, id FROM package_notes WHERE release = ''""")
- if release:
+ if release != 'sid':
c.execute("""INSERT OR REPLACE INTO vulnlist
SELECT bug_name, package, id FROM package_notes
WHERE release = ?""", (release,))
- else:
- release = 'sid'
urgency_to_flag = {'low' : 'L', 'medium' : 'M', 'high' : 'H',
'not yet assigned' : ' '}
@@ -1732,7 +1735,7 @@ class DB:
store_value('release/1/' + release, '\n'.join(result))
- for release in ('sid', 'jessie', 'stretch', 'buster', 'bullseye'):
+ for release in config.get_supported_releases():
gen_release(release)
result = result_start
@@ -1745,7 +1748,7 @@ class DB:
def calculateDebsecan(self):
"""Calculate all debsecan data."""
- for release in ('', 'jessie', 'stretch', 'buster', 'bullseye'):
+ for release in config.get_supported_releases():
self.calculateDebsecan0(release)
self.calculateDebsecan1()
@@ -1778,13 +1781,16 @@ class DB:
"""A generator which returns tuples (RELEASE-LIST, VERSION),
the available versions of the source package pkg."""
+ releases = config.get_supported_releases()
+ values = [pkg] + releases
+
for (release, version) in cursor.execute(
"""SELECT release_name(release, subrelease, archive)
AS release, version FROM source_packages
WHERE name = ?
- AND release IN ('jessie', 'stretch', 'buster', 'bullseye', 'sid')
+ AND release IN (""" + ",".join("?" * len(releases)) + """)
GROUP BY release, version
- ORDER BY release_to_number(release), subrelease_to_number(subrelease), version COLLATE version""", (pkg,)):
+ ORDER BY release_to_number(release), subrelease_to_number(subrelease), version COLLATE version""", values):
yield release, version
def getBinaryPackageVersions(self, cursor, pkg):
@@ -1830,6 +1836,9 @@ class DB:
RELEASE-LIST, VERSION, VULNERABLE-FLAG) of source packages
which are related to the given bug."""
+ releases = config.get_supported_releases()
+ values = [bug] + releases
+
for (package, releases, version, vulnerable) in cursor.execute(
"""SELECT package, string_list(release), version, vulnerable
FROM (SELECT p.name AS package,
@@ -1837,10 +1846,10 @@ class DB:
p.version AS version, s.vulnerable AS vulnerable
FROM source_package_status AS s, source_packages AS p
WHERE s.bug_name = ? AND p.rowid = s.package
- AND release in ('jessie', 'stretch', 'buster', 'bullseye', 'sid'))
+ AND release in (""" + ",".join("?" * len(releases)) + """))
GROUP BY package, version, vulnerable
ORDER BY package, releasepart_to_number(release), subreleasepart_to_number(release), version COLLATE version""",
- (bug,)):
+ values):
yield package, releases.split(', '), version, vulnerable
def getBugsFromDebianBug(self, cursor, number):
@@ -2026,59 +2035,6 @@ class DB:
ORDER BY n.package"""):
yield (package, bugs.split(','), map(int, debian_bugs.split(',')))
- def getEffectiveVersion(self, release, pkg, purpose, cache=None, cursor=None):
- """Retrieve the effective version of a source package in a release.
-
- The effective version is the version that matches the recommended
- sources.list file for the intended purpose. For suitable values
- of purpose, see dist_config.
- """
- # The cache is structured as a (RELEASE, PACKAGE) => VAL
- # dict, where VAL is either a dict PURPOSE => VERSION,
- # a VERSION, or None.
- if cache is not None:
- sp = (release, pkg)
- if sp in cache:
- d = cache[sp]
- if d.__class__ == dict:
- return d.get(purpose, None)
- else:
- return d
-
- if cursor is None:
- cursor = self.cursor()
-
- rel = dist_config.releases[release]
- purposes = rel['purpose']
- results = {}
-
- Version = debian_support.Version
- for (part, ver) in cursor.execute(
- """SELECT DISTINCT subrelease, version FROM source_packages
- WHERE release = ? AND name = ?""", (str(release), pkg)):
- ver = Version(ver)
- for (purpose, permitted) in purposes.items():
- if part not in permitted:
- continue
- if purpose in results:
- oldver = results[purpose]
- if ver <= oldver:
- continue
- results[purpose] = ver
-
- if cache is not None:
- vers = set(map(str, results.values()))
- l = len(vers)
- if l == 1:
- for r in vers:
- cache[sp] = Version(r)
- elif l == 0:
- cache[sp] = None
- else:
- cache[sp] = results
-
- return results.get(purpose, None)
-
def check(self, cursor=None):
"""Runs a simple consistency check and prints the results."""
View it on GitLab: https://salsa.debian.org/security-tracker-team/security-tracker/-/compare/5691f3044f636bfacb4b1ec5960a290d105cef59...12052760d6f12fd9ec6e3c42afe66449011b01cc
--
View it on GitLab: https://salsa.debian.org/security-tracker-team/security-tracker/-/compare/5691f3044f636bfacb4b1ec5960a290d105cef59...12052760d6f12fd9ec6e3c42afe66449011b01cc
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/debian-security-tracker-commits/attachments/20200604/822e7f99/attachment-0001.html>
More information about the debian-security-tracker-commits
mailing list