Bug#724610: Acknowledgement (incrementally updated databases eventually become corrupt)

[Joey Hess]

I was able to reproduce some database corruption, using the attached
foo.pl script. 

I ran it once to create the xapian db, and ctrl-c'd after a while. 
xapian-check was happy.

Then I ran it in an evil loop:

while xapian-check foo; do perl foo.pl  & sleep 1m; kill -9 %1; done

After about an hour, xapian-check found that the database had indeed
become corrupt:

hjoey at pell:~/foo>xapian-check .
record:
baseA blocksize=8K items=2580000 lastblock=27242 revision=258 levels=2 root=27030
B-tree checked okay
record table structure checked OK

termlist:
baseA blocksize=8K items=2580000 lastblock=20465 revision=258 levels=2 root=20303
B-tree checked okay
termlist table structure checked OK

postlist:
baseB blocksize=8K items=5610265 lastblock=54544 revision=259 levels=2 root=8817
B-tree checked okay
document id 2580001 in doclen stream is larger than get_last_docid() 2580000
document id 2580001: length 14 doesn't match 0 in the termlist table
document id 2580002 in doclen stream is larger than get_last_docid() 2580000
document id 2580002: length 14 doesn't match 0 in the termlist table
document id 2580003 in doclen stream is larger than get_last_docid() 2580000
document id 2580003: length 14 doesn't match 0 in the termlist table
document id 2580004 in doclen stream is larger than get_last_docid() 2580000
document id 2580004: length 14 doesn't match 0 in the termlist table
document id 2580005 in doclen stream is larger than get_last_docid() 2580000
document id 2580005: length 14 doesn't match 0 in the termlist table
...
postlist table errors found: 20000

position:
baseB blocksize=8K items=12949980 lastblock=35161 revision=259 levels=2 root=34883
B-tree checked okay
position table structure checked OK


This is different corruption than I have seen from ikiwiki, but it
certianly suggests that relying on the automatic flush at exit is not
safe if there is any chance that the script can be killed. (Or the machine
unexpectedly lose power?)

-- 
see shy jo
-------------- next part --------------
A non-text attachment was scrubbed...
Name: foo.pl
Type: text/x-perl
Size: 638 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/pkg-perl-maintainers/attachments/20130925/9e26c5b3/attachment.pl>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 828 bytes
Desc: Digital signature
URL: <http://lists.alioth.debian.org/pipermail/pkg-perl-maintainers/attachments/20130925/9e26c5b3/attachment.sig>