[Teammetrics-discuss] Incomplete imports

[Andreas Tille]

Hi Sukhbir,

the run with the code from 8.8.2011 to import *all* nntp lists finished
over night.  It resulted in the following entries in the database:

teammetrics=# SELECT project, count(*) from listarchives where domain='lists.debian.org' group by  project;
      project       | count
--------------------+-------
 debian-edu         |     4
 debian-openoffice  | 24812
 debian-qa          |  1212
 debian-lex         |   159
 debian-derivatives |   552
 debian-science     |  4371
 debian-med         |  7551

If I remember correctly these few were just inside the database before I
started the whole import (but even these are to view considering that it
was run with

   blends.debian.net:/etc/teammetrics/listinfo.conf_short

which was used for a shorter test.  If you look at the current active
config file there should be about 30 lists in the database.  We have

$ ls /var/cache/teammetrics/*.mbox
/var/cache/teammetrics/debian-accessibility-1-3836.mbox
/var/cache/teammetrics/debian-amd64-2-27593.mbox
/var/cache/teammetrics/debian-amd64-27594-27596.mbox
/var/cache/teammetrics/debian-arm-10188-10188.mbox
/var/cache/teammetrics/debian-arm-10189-10190.mbox
/var/cache/teammetrics/debian-arm-10191-10201.mbox
/var/cache/teammetrics/debian-arm-1-10187.mbox
/var/cache/teammetrics/debian-blends-1-3212.mbox
/var/cache/teammetrics/debian-boot-1-131020.mbox
/var/cache/teammetrics/debian-boot-131021-131034.mbox
/var/cache/teammetrics/debian-boot-131035-131038.mbox
/var/cache/teammetrics/debian-boot-131039-131048.mbox
/var/cache/teammetrics/debian-ctte-1-2618.mbox
/var/cache/teammetrics/debian-curiosa-1-4495.mbox
/var/cache/teammetrics/debian-derivatives-1-555.mbox
/var/cache/teammetrics/debian-derivatives-556-557.mbox
/var/cache/teammetrics/debian-desktop-1-2944.mbox
/var/cache/teammetrics/debian-devel-1-164399.mbox
/var/cache/teammetrics/debian-devel-announce-1-1616.mbox
/var/cache/teammetrics/debian-devel-games-1-5092.mbox
/var/cache/teammetrics/debian-devel-games-5093-5094.mbox
/var/cache/teammetrics/debian-edu-1-23436.mbox
/var/cache/teammetrics/debian-edu-23437-23441.mbox
/var/cache/teammetrics/debian-edu-23442-23442.mbox
/var/cache/teammetrics/debian-embedded-1-6753.mbox
/var/cache/teammetrics/debian-firewall-1-6306.mbox
/var/cache/teammetrics/debian-gis-1-178.mbox
/var/cache/teammetrics/debian-i18n-1-15443.mbox
/var/cache/teammetrics/debian-isp-1-17239.mbox
/var/cache/teammetrics/debian-jr-1-1690.mbox
/var/cache/teammetrics/debian-jr-1691-1691.mbox
/var/cache/teammetrics/debian-kde-1-25749.mbox
/var/cache/teammetrics/debian-kernel-1-73304.mbox
/var/cache/teammetrics/debian-l10n-german-1-17672.mbox
/var/cache/teammetrics/debian-laptop-1-24530.mbox
/var/cache/teammetrics/debian-legal-1-33931.mbox
/var/cache/teammetrics/debian-lex-1-166.mbox
/var/cache/teammetrics/debian-lex-167-167.mbox
/var/cache/teammetrics/debian-live-1-11692.mbox
/var/cache/teammetrics/debian-med-1-7795.mbox
/var/cache/teammetrics/debian-med-7796-7802.mbox
/var/cache/teammetrics/debian-med-7803-7810.mbox
/var/cache/teammetrics/debian-mentors-1-50528.mbox
/var/cache/teammetrics/debian-mips-1-7131.mbox
/var/cache/teammetrics/debian-multimedia-1-3791.mbox
/var/cache/teammetrics/debian-multimedia-3792-3792.mbox
/var/cache/teammetrics/debian-newmaint-1-5959.mbox
/var/cache/teammetrics/debian-ocaml-maint-1-20058.mbox
/var/cache/teammetrics/debian-ocaml-maint-20059-20065.mbox
/var/cache/teammetrics/debian-openoffice-1-27829.mbox
/var/cache/teammetrics/debian-openoffice-27830-27851.mbox
/var/cache/teammetrics/debian-perl-1-9362.mbox
/var/cache/teammetrics/debian-policy-1-15206.mbox
/var/cache/teammetrics/debian-project-1-19391.mbox
/var/cache/teammetrics/debian-python-1-7066.mbox
/var/cache/teammetrics/debian-qa-1-11407.mbox
/var/cache/teammetrics/debian-qa-11408-11408.mbox
/var/cache/teammetrics/debian-release-1-47154.mbox
/var/cache/teammetrics/debian-russian-1-106613.mbox
/var/cache/teammetrics/debian-science-1-4481.mbox
/var/cache/teammetrics/debian-science-4482-4484.mbox
/var/cache/teammetrics/debian-science-4485-4496.mbox
/var/cache/teammetrics/debian-security-1-16439.mbox
/var/cache/teammetrics/debian-testing-1-6048.mbox
/var/cache/teammetrics/debian-user-1843-415150.mbox
/var/cache/teammetrics/debian-women-1-4328.mbox
/var/cache/teammetrics/debian-women-4329-4329.mbox

$ ls /var/cache/teammetrics/*.mbox | wc -l
67

Any clue why those mboxes are not injected into the database?

This somehow proves my point that the Algorithm

  1. Download single list
  2. Inject data into database
  3. Register list as injected
  4. Goto next list

might make sense.

Kind regards

      Andreas.

-- 
http://fam-tille.de