[Blends-commit] [Git][blends-team/blends][master] Description how Blends relevant data are gathered and stored

[Andreas Tille]

Andreas Tille pushed to branch master at Debian Blends Team / blends


Commits:
9c630f3d by Andreas Tille at 2018-04-03T16:56:53+02:00
Description how Blends relevant data are gathered and stored

- - - - -


1 changed file:

- doc/en/A_devel.xml


Changes:

=====================================
doc/en/A_devel.xml
=====================================
--- a/doc/en/A_devel.xml
+++ b/doc/en/A_devel.xml
@@ -911,4 +911,54 @@ the changes by doing:
 </orderedlist>
 </para>
 </sect1>
+  <sect1 id="datagathering">
+  <title>Description how Blends relevant data are gathered and stored</title>
+<para>
+All data relevant for Blends are available in
+<ulink url="https://wiki.debian.org/UltimateDebianDatabase">Ultimate Debian Database (UDD)</ulink>.
+Here is a description of the data gatherers that were explicitly invented to provide
+information for usage in Blends websentinel.
+</para>
+
+  <sect2 id="ftpnew">
+  <title>Packages in Debian ftp new queue</title>
+<para>
+New packages might reside for some time in the so called new queue until
+a member of the ftpmaster team has evaluated the package as fit for the
+Debian distribution.  To enable informing our users about that status the
+new queue is parsed and the information displayed in the web sentinel pages.
+The actual gatherer code can be found in UDD Git and is split into a
+<ulink url="https://salsa.debian.org/qa/udd/blob/master/scripts/fetch_ftpnew.sh">Script that fetches the data</ulink>
+and a <ulink url="https://salsa.debian.org/qa/udd/blob/master/udd/ftpnew_gatherer.py">Parser which injects the data into UDD</ulink>.
+</para>
+  </sect2>
+
+  <sect2 id="machine_readable">
+  <title>Machine readable data in Git repositories of Blends and some packaging teams</title>
+<para>
+Several machine readable data of packages are parsed from Git repositories.
+There are data that might be more up to date in Git (for instance scientific
+publication data or some registry entries).  There are also data for not
+yet available packages a Blends team is working on.  To enable refering to
+this data some selected Git repositories are parsed.  The parser for Salsa
+is available in the
+<ulink url="https://salsa.debian.org/blends-team/website/blob/master/misc/machine_readable/fetch-machine-readable_salsa.py">website Git</ulink>
+of the Blends project.  It is running in a daily cron job on the host
+<ulink url="http://blends.debian.net">blends.debian.net</ulink> and
+creates an <ulink url="http://blends.debian.net/_machine-readable/machine-readable.tar.xz">archive</ulink>
+with all the machine readable files found in the specified Git repositories.
+</para>
+<para>
+This <ulink url="http://blends.debian.net/_machine-readable/machine-readable.tar.xz">archive</ulink>
+is read into UDD by an
+<ulink url="https://salsa.debian.org/qa/udd/blob/master/udd/blends_prospective_gatherer.py">UDD
+importer script</ulink> in a cron job which is totally independent from the job
+that collects the data from Salsa.  Due to the disconnected jobs running on
+different hosts there might be some undetermined delay between changing
+some metadata in a Git repository and the time when the data are available
+in UDD.
+</para>
+  </sect2>
+
+</sect1>
   </appendix>



View it on GitLab: https://salsa.debian.org/blends-team/blends/commit/9c630f3d4a2facae7d75e44f905ebb195b897dda

---
View it on GitLab: https://salsa.debian.org/blends-team/blends/commit/9c630f3d4a2facae7d75e44f905ebb195b897dda
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/blends-commit/attachments/20180403/1cda8f0e/attachment-0001.html>