Migrating from Cyrus to Exchange and from mbsync -> offlineimap in the process

[Ævar Arnfjörð Bjarmason]

At work we recently migrated from Cyrus IMAP to a MS Exchange server.
I had to figure out how to migrate my Maildir's syncing. I ended up
moving from mbsync to offlineimap because I couldn't make the former
work.

I'm posting this to both relevant mailing lists mainly so that anyone
who searches for the same error message I was getting will have an
easier time.

I have a Maildir of around 20GB / 700K messages.

When the IMAP server got moved the mail all got moved as well, so I
was hoping that I wouldn't need a resync. I was using mbsync at the
time.

First thing I tried:

    1. rsync -av --progress ~/Maildir/ ~/Maildir.2016-07-11
    2. find ~/Maildir -iname '*state*' -exec rm -v {} \;
    3. until mbsync --verbose --pull work; do echo "Failed at
$(date)"; sleep 1; done

That didn't work, it initially synced like ~500 messages, but then it
seems to spend hours initially updating the journal and then get this:

    Warning: lost track of 925645 pulled message(s)
    Socket error: write to <ip> (<ip>:<port>): Connection reset by peer

And subsequently because I was running this in a loop the command took
~20 minutes but then always failed with:

    slave: 577558 messages, 443643 recent
    Socket error: write to <ip> (<ip>:<port>): Connection reset by peer

I then did a second try based on
https://sourceforge.net/p/isync/mailman/message/31260886/ ("Error:
UIDVALIDITY of master change"):

    perl -pi -e 's/MasterUidValidity \d+/MasterUidValidity 613272/g'
$(find Maildir -name '*mb*')

But I always kept getting new UIDs:

    Error: UIDVALIDITY of master changed (got 597077, expected 613272
    Error: UIDVALIDITY of master changed (got 613278, expected 613272

Finally with mbsync I forgot about syncing on top of the old Maildir
and just tried a fresh sync:

    until mbsync --verbose --pull work; do echo "Failed at $(date)";
sleep 1; done

This got me around 500 messages and then:

    C: 0/1  B: 0/5  M: +0/0 *0/0 #0/0  S: +500/958585 *0/0 #0/0
    Socket error from <ip> (<ip>:<port>): Connection reset by peer
    C: 1/1  B: 1/5  M: +0/0 *0/0 #0/0  S: +500/958585 *0/0 #0/0

And on the next run:

    slave: 500 messages, 0 recent
    Warning: lost track of 958085 pulled message(s)

It then just seemed to hang, and strace(1)-ing it shows nothing.

I then gave up and went for offlineimap again. I'd initially used
that, but moved to mbsync because offlineimap had no option to ignore
deletes on the remote. But since then this option has been
implemented: https://github.com/OfflineIMAP/offlineimap/issues/47

Moving to offlineimap worked right away. It's doing something
different with the IMAP protocol that works with Exchange. Instead of
hanging for a long time until the server seems to disconnect the
connection it downloaded all the messages in very short order.

Summary:

 * Moved from Cyrus to Exchange

 * Had to move from mbsync to offlineimap in the process

 * pros: offlineimap is more configurable in some ways. Not in others.
I miss the "Sync Push/Pull <x>" configuration in mbsync

 * pros: It seems to work better with the Exchange IMAP protocol

 * cons: offlineimap takes up a *lot* more resources than mbsync.
Around ~120% CPU. I don't remember mbsync ever taking more than 10-20%
CPU. I just run it in a full loop though and haven't tried out IDLE
etc. so maybe I'm using it wrong.

 * cons: Getting mail with offlineimap takes a lot longer, around 2
minutes to just find out that it needs to do very little. Took around
10-20 seconds with mbsync, although this was on Cyrus, and I have
around 700K messages in my INBOX. So it may be Exchange that's slower,
I'm not sure.