hello and starting using offlineimap in china

[Luke Kenneth Casson Leighton]

hi, had to set up offlineimap. and cyrus.  and prayer webmail.  and
exim4-lmtp.  and exim4 authenticated smtp proxying...

...because i'm currently operating in china and it's a bitch.  constant
HTTP and especially HTTPS related timeouts (SSL handshake errors
mostly)... maan it's insane.

so, hello, out of necessity i'll be patching offlineimap so that
i can actually like... communicate effectively by email.

i'm currently in HK and will be in Taiwan next, so have decent
connectivity for a while before i return to Zhuhai, i hope to
have everything i need set up over the next four to five weeks.

nicolas, thank you for agreeing an exception to the policy of
not communicating publicly whilst i was exploring how to get
in touch (and to send patches), i'm fine for the next 4-5 weeks
and also have sync'd up (250,000 messages!!) really quick using
the HK airport hotel's WIFI.  what would have taken 5 days @
1 message per 2-5 seconds was instead 4 messages / second and
completed overnight.  whew :)

i have two patches i'll send separately then i want to work
on a third.

first: adding in authproxy config option.  turns out that china
doesn't interfere with imap.gmail.com (yay!) but it *does*
completely balls up accounts.google.com which makes it impossible
to use OAUTH2.  so, i *only* need a proxy for oauth2 *not* IMAP,
which means adding in a separate option.

second: retries.  god is it insane to be operating at 6-20k/sec
with 20 to 80% packet loss and ping times that vary between 385ms
and a whopping 40000ms.  over openvpn when using tcp i've actually
seen ping times of 200 *seconds*.  staggering.  anyway, SSL
handshakes timeout, SSL connections return EOF on a constant and
regular basis, the TCP stack gets its knickers in a twist also
on a regular basis... simplest thing to do: instead of defaulting
to 2 retries and terminating, allow an arbitrary number of retries
otherwise if you just put offlineimap into a while loop it just
totally wastes CPU cycles reloading the sqlite database.

third: with 250,000 messages in the inbox, offlineimap.py ends up
using 100% CPU for a good couple of minutes JUST loading the sqlite
database.  i've looked at the sqlite code: there's only one
table, it's effectively a key-value store... i'm a huge fan of
lmdb and especially python-lmdb... so i'll be adding that as an
option.  i can't possibly tolerate this laptop going into 100%
CPU usage for up to 100 seconds at a time.  lmdb is aawesome, i
studied it whilst writing up the wikipedia page, and used it for
a real-time network traffic parsing system to handle tens of
thousands of packets per second... in python of all languages.

anyway, thank you nicolas for explaining (in private email) about
the DCO http://www.offlineimap.org/doc/dco.html
i don't know if you need me to write explicitly here that i
agree to it, but yes please take this as indication that i do
indeed agree to it and certify that all contributions conform
to the DCO.  i'll be using git commit -s in future.

ok patches to follow for 1 and 2

thanks to everyone who's contributed, for creating some extremely
necessary software.

l.

-- 
--
-- http://crowdsupply.com/eoma68 eco-conscious computing