[debian-mysql] Groping for a roadmap in the dark

[Nicholas Bamber]

There are currently three major differences between Ubuntu and Debian's 
repository:
1.) Ubuntu uses upstart rather than init.
upstart looks like great and I would like Debian to support it. However 
currently
it is not available in on non-Linux and I doubt it will ever be the default.
Therefore Debian must find a way of working with upstart if it is in use 
and using traditional init otherwise. If we can achieve that I do not see
why ubuntu should require any customizations but that would preclude
mysql-server-5.5 depending on upstart as it does in ubuntu.
2.) I have no issue with the apparmor stuff in principle. I am still trying
to work my way through the documentation. However until the other two issues
look more tractable I fail to see resolving this as  a high priority.
3.) The packet restructuring (mysql-client-core-5.5, mysql-testsuite) - 
The rest of this email is really about that.

Principles of Good Package Structure
------------------------------------
I cannot profess to understand good package structure but I am pretty sure
we don't have it. So I am groping to find what that would be.

1.) Piuparts is a statement of how a Debian package should behave. 
Essentially
a package should not stomp over its own or other packages and should 
clean up
after itself when asked (i.e. when asked to do a purge). We simply don't
comply at the moment. Given Debian MySQL's bad history on this we will 
have to take extra care but I feel it has to be done.

2.) The client/server divide. There should be a clear divide between the two
(apart from mysql-common). They should be happy to coexist on the same 
machine
or be on separate machines and it should be possible to upgrade one and 
not the
other.  A client side executable is one that only needs
a network connection to communicate with the server. A server side 
machine needs
to be on the same machine. Again the current package does not conform.

3.) A smooth upgrade path. We had issues this time. I am trying to 
understand what the theory is.

a.) mysql-common should always be backwards compatible (not indefinitely 
but for
for the most recent previous major version). This can be achieved by 
bringing in
new options as "loose" options. When we move them to non-loose we put in 
Breaks
clauses the
old package. This would allow, say, the client side to be upgraded 
whilst the server side was kept back. The way it was done this time was 
defensible but went
because of other issues. To adopt this policy would be regarded as defensive
package management.

b.) I think the packages can be usefully divided into three "slices": 
unversioned,
versioned and build-time. The versioned packages (libmysqlclientS, 
mysql-server-core-N, etc) are used in run-time. The unversioned packages 
are very light or virtual packages that provide some naming stability 
across major versions.
Build-time packages being unversioned means you can only build against 
the latest version. Postgresql puts the unversioned packages in a 
seperate source
package. This allows several versions to coexist (but you can only build 
against
one) but at least the client and server tools could coexist. However the 
more
I thought about this (and talked to the release team a little), the less
I can see that the Postgresql model is better.

c.) Another useful way thinking about the packages is "stacks". We have the
following stacks:
     server: mysql-server-core-N, mysql-server-N, mysql-server
     client: mysql-client-core-N, mysql-client-N, mysql-client     test: 
mysql-testsuite-core-N, mysql-testsuite-N, mysql-testsuite
         [ only a semi-stack because it has to have the same version as 
server ]
     source: really on its own
     library: libmysqlclientS, libmysqlclient-dev, libmysqld-dev & 
libmysqld-pic
A stack is a grouping of packages that can be thought of as a unit - and a
way of breaking up the relationships between packages into more manageable
chunks. So the dependencies between stacks should be as minimal and
as loosely coupled as possible.
So what should happen during a major version upgrade? Well mysql-common 
should
always be backwards compatible so that upgrade just goes through. The
*unversioned* top of stack X tries to upgrade - but will have 
dependencies on
the new package X-N just below - and X-N will have Breaks/Replaces 
clauses against
X-P (where P<N). So the whole sequence is:
i.) Version N of X gets downloaded and uppacked
ii.) However it has an unfulfilled dependency on X-N so that gets downloaded
and unpacked.
iii.) X-N however has Breaks/Replaces against X-P so X-P gets deconfigured.
iv.) X-N can now get configured and installed
v.) X-P now gets removed
vi.) Version N X can now get installed.

4.) As noone has expressed any strong opinions I am proposing to split the
mysql-server-N and mysql-client-N packages into the following:
     innotop - separate source, mysql-client-N will depend on and later 
recommend it
     mysql-myisam-utils - those server side utilities associated with MyISAM
     mysql-server-utils - non-essential server side utilities
     text-replace - the replace program, needed by mysql-myisam-utils

5.) I think it would be good to have the mysql packages sharing a docs 
directory.
It would save space and actually would be logical.  For technical
reasons the base package has to be arch:any. mysql-common is not a good 
choice
actually because it contains the my.cnf and it is a good idea for 
mysql-client-core-N, mysql-server-core-N and mysql-source-N not to have that
automatically. So the proposal would be that:
i.) The magic arch:any package containing the copyright, and upstream 
and Debian
changelogs would be called mysql-N (i.e. the same as the source package).
ii.) mysql-common would be renamed mysql-config, but would still Provide
mysql-common,  and would depend on mysql-N so it would look a lot like the
current mysql-common. It would also contain a README.Debian explaining
the reasoning behind the new structure.

6.) Config modularity. The modern way would
be for mysql-config to store a skeleton /etc/mysql/my.cnf and something 
to keep
/etc/mysql/conf.d alive. However we will have to keep just enough
in /etc/mysql/my.cnf not to break the existing 5.5. Each of the packages
would own its appropriate config fragment below /etc/mysql/conf.d .

7.) We use pkg-kde-tools to do the right thing by the shlibs system.

8.) I said that the dependencies between stacks should be minimized.
i.) the client stack has dependencies on the library package because of
the Perl programs. If the Perl scripts were in a seprate package that
was only Recommended this dependency would be loosened.
ii.) The server stack depends on the client stack because of check scripts
that run during start up. With a bit of analysis this could probably be 
tightened
but at the very least I see no reason to require an exact match of minor 
version
across this interface.
iii.) Obviously the test suite depends fundamentally and extensively on 
the client
and server (and maybe even library) stacks.

9.) I have tried to put together a diagram of how I envisage the new package
structure to look:
http://www.periapt.co.uk/img/arcana/mysql.png

It tries to express visually what I mean by stacks.

I have a cunning plan
---------------------
1.) We talk about it. If noone discusses this stuff with me
then I shall just go and implement it - albeit in experimental.
2.) One issue I know will get a discussion - how to acheive piuparts 
compliance.
I favour unconditional compliance. That means that, after every conceivable
warning and opportunity to back up data and a NEWS item on upgrade, if the
user says 'apt-get purge' (note purge - not  a mere 'remove') then 
/var/lib/mysql
gets nuked.  On the other hand Olaf has volunteered to talk to the 
piuparts team
about whether MySQL (and presumably other databases) get an exception. 
However
before he does that, I think we should guage how much consensus there is on
the mailing list over the issue.
3.) Once there is a consensus over the package restructuring I would
feel more comfortable about bringing Debian into line with Ubuntu on the 
mysql-client-core-5.5 and mysql-testsuite changes.
4.) I will build a test framework to automate as much testing described in
http://wiki.debian.org/Teams/MySQL/PreuploadTests  as possible. In 
particular
this means automated building of reverse dependencies.
5.) I implement the package restructuring described above (subject to the
outcome of any discussions).
6.) Then based upon Olaf's discussions with the piuparts team,
I will implement piuparts compliance and fix any bugs that are relevant.
7.) At this point we could play with MariaDB and Percona as drop in 
replacements
for mysql-server-core-N.
8.) By this point I expect to have played with apparmor in Debian to have
backported the Ubuntu changes to unstable (as long as its post freeze). Of
course I would have no issue with Clint doing so in the meantime.
9.) By this point I also expect to have played with upstart. However I never
expect upstart to be default in Debian, and if the package is to be 100%
synchronized between Ubuntu and Debian, then there will have to be 
changes on the Ubuntu side. Also there are issues with upstart in Debian 
currently.
10.) We use pkg-kde-tools and fix all the library issues. #673574 etc
11.) By this point I would expect the package to be lintian clean.
12.) By this point I would expect 5.6 to have been released. We would then
give a heads up to the Debian release team.
13.) Then we would upload 5.6 to experimental and start raising bugs.