[PATCH/RFC 1/7] New feature: Update folder structure to new config parameters.

[Hubert Pineault]

Nicolas Sebrecht writes:

> On Sun, Mar 03, 2019 at 03:56:57AM -0500, Hubert Pineault wrote:
>
>>   Motivation:
>>     If you change some of the config file paremeters, like nametrans
>>     and utf8 decode, it will completly mess your maildir folder
>>     structure. So, to make changes in config file, you need to
>>     re-download the whole imap account.
>
> Correct.
>
>>                                         If it weights a few gigs,
>>     it's a bit annoying. With the coming of features like remote
>>     folder creation and utf8 decoding of folder names, a converting
>>     tool could be handy for some of us.
>
> You're trying to address two different goals:
>
> 1. gently moving to utf8 support
> 2. support nametrans updates

This distinction is interesting. Although, it doesn't exactly apply to
this patch.

Originally, I started this project because I had switched from an
offlineimap version not supporting utf8foldername and reverse nametrans
to the lastest version. I wanted to be able to apply utf8 names and the
nametrans I was using on the old version were not thought to be
reversible. More over, my email account was polluted by other email
client, tests, etc. So I thought, it'd be a good idea to write a program
to change folder name and merge folders. I was half way through and I
thougth, hey why not try to do this from the source code. I'll send both
config in an other mail to give you a better idea.

>
> For (1) utf8, you're trying to address
> https://github.com/OfflineIMAP/offlineimap/issues/501
> which is not a small one. Great!
> I've just updated this issue, BTW.

The way I see it. I'm not really adressing this issue. Well, I'm only
doing half the job.
What you mention in the discussion is three points:
  1) The need to remember utf7 name from imap list command even if utf8
  is enabled. This is what I'm doing in patch 5, for IMAPRepository.
  2) To make the same thing with MaildirFolder and add a function to
  make the conversion.
  3) The main difficulty is to consider those two names as synonymes in
  appropriate getfolders() methods. This patch is not dealing with this.

>
> So, we might rather like to only support moving to UTF8 without going
> back. In this case, we could introduce your new policy: create a new
> config file, metadatadir and maildirs while avoiding re-downloads and
> update everything is needed (md5, etc).

With my code, issue 501 becomes rather useless, since it takes less than
10 sec to run and can switch from utf8-utf7 and utf7-utf8.

>
> Something like --convert-to-utf8 could help to keep the code as simple
> as possible. This way, the code about this UTF conversion might be moved
> out from the syncing logic.

This could be the subject of a first pull request. Although, I think
this should be made having in mind a more complexe conversion process to
include nametrans in a second pull request. So, I think we should take
time to discuss my code before getting to this.

>
>
> About (2) nametrans, we must take care about what we do. I still don't
> understand how we would correctly map the old translated foldernames
> to the new translated foldernames because it's tracked anywhere.
>
> Suppose we have a IMAP folder "A" and there is a nametrans translating
> "A" to "J". When we write a new nametrans translating "A" to "X", how do
> we know that "X" is the new "J"? The only use case we know "X" is the
> new "J" is when moving to UTF-8. BTW, this conversion can break the
> nametrans by other means because matching rules for strings in UTF-7 and
> UTF-8 can be distinct. I think this operation likely requires manual
> checks from the user and we can't fully automate the process.
>

This is exactly the goal of this patch. Using you exemple, to link "J"
to "X", we need to use "A". This is why we need to remember utf7 folder
names even if utf8foldernames is enabled. This is adressed in patch 7.
The update process handles the merging of folders if the two version of
nametrans make it so. This been said, I suggest that we hold conversion on
this subject and adress it under patch 5 message.

>
>>   Introduction:
>>     The objectif is to convert an actual maildir folder structure to a
>>     new one. There is still a farely good amount of work to do in
>>     order to make it ready to merge with main branch. I have already
>>     identified some changes that should be made in the update
>>     process. Those points are raised in details related commits'
>>     messages. I will need some help with cleaning the code and,
>>     notably, exceptions handling.
>>
>>     The three main points that will need a complete rewritting are:
>>
>>       - The invoking method (actually, we load two independent config
>>         files)
>>       - The way different config parameters are handled by objects
>>       - (actually, we use two accounts, so two remote, local and status
>>         repos)
>>       - Instead of altering existing class, it could be a better idea to
>>         create a child class specific for updating.
>
> My first feeling after a quick review of this path series is that we
> might really like to keep things as simple as possible. Currently,
> the required changes are deep with a lot of duplication of code.
>
> Converting the config file, metadata and maildirs from UTF-7 to UTF-8
> could be one way and non-reversible. This would be fine if we agree on
> your approach of duplicating all the local content to a full new setup.
> The other good news is that I guess we could keep the sync and
> conversion logic seperated. I think this would help us to keep the code
> much more simple.


--
Hubert Pineault
https://github.com/hubearth/offlineimap