[debian-mysql] Groping for a roadmap in the dark

[Clint Byrum]

Excerpts from Nicholas Bamber's message of 2012-06-03 15:48:39 -0700:
> There are currently three major differences between Ubuntu and Debian's 
> repository:
> 1.) Ubuntu uses upstart rather than init.
> upstart looks like great and I would like Debian to support it. However 
> currently
> it is not available in on non-Linux and I doubt it will ever be the default.
> Therefore Debian must find a way of working with upstart if it is in use 
> and using traditional init otherwise. If we can achieve that I do not see
> why ubuntu should require any customizations but that would preclude
> mysql-server-5.5 depending on upstart as it does in ubuntu.

There is a policy change needed to make upstart viable in Debian. If you
feel that upstart should be in Debian (I know I do) then please go here
and voice your opinion:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=591791

> 2.) I have no issue with the apparmor stuff in principle. I am still trying
> to work my way through the documentation. However until the other two issues
> look more tractable I fail to see resolving this as  a high priority.

The delta for Ubuntu to have this is pretty tiny, so I don't think its
too important to worry about this for that reason. As AppArmor gets
better in Debian, we should find it easy to turn it on though. Lets
shelve this until other things are worked out.

> 3.) The packet restructuring (mysql-client-core-5.5, mysql-testsuite) - 
> The rest of this email is really about that.
> 
> Principles of Good Package Structure
> ------------------------------------
> I cannot profess to understand good package structure but I am pretty sure
> we don't have it. So I am groping to find what that would be.
> 
> 1.) Piuparts is a statement of how a Debian package should behave. 
> Essentially
> a package should not stomp over its own or other packages and should 
> clean up
> after itself when asked (i.e. when asked to do a purge). We simply don't
> comply at the moment. Given Debian MySQL's bad history on this we will 
> have to take extra care but I feel it has to be done.
> 

Piuparts is an awesome assessment of where we stand w.r.t. policy. Its true,
we are not automatically purging data if the question about whether or not
to remove /var/lib/mysql/* is left to defaults.

I'd entertain making mysql-server-5.5/postrm_remove_databases default to true
rather than false, as long as it is noted in NEWS, and the question is given
a high priority. As I understand it, this would solve the issue entirely.

> 2.) The client/server divide. There should be a clear divide between the two
> (apart from mysql-common). They should be happy to coexist on the same 
> machine
> or be on separate machines and it should be possible to upgrade one and 
> not the
> other.  A client side executable is one that only needs
> a network connection to communicate with the server. A server side 
> machine needs
> to be on the same machine. Again the current package does not conform.

All good ideas, but I don't know that we need to do anything to change
this.  What specifically is overlapping here?

> 
> 3.) A smooth upgrade path. We had issues this time. I am trying to 
> understand what the theory is.
> 
> a.) mysql-common should always be backwards compatible (not indefinitely 
> but for
> for the most recent previous major version). This can be achieved by 
> bringing in
> new options as "loose" options. When we move them to non-loose we put in 
> Breaks
> clauses the
> old package. This would allow, say, the client side to be upgraded 
> whilst the server side was kept back. The way it was done this time was 
> defensible but went
> because of other issues. To adopt this policy would be regarded as defensive
> package management.

Yeah I didn't know we had the option of the 'loose' options, this will
allow us to more smoothly transition on major version updates. This
seems like a no-brainer, lets do that.

> 
> b.) I think the packages can be usefully divided into three "slices": 
> unversioned,
> versioned and build-time. The versioned packages (libmysqlclientS, 
> mysql-server-core-N, etc) are used in run-time. The unversioned packages 
> are very light or virtual packages that provide some naming stability 
> across major versions.
> Build-time packages being unversioned means you can only build against 
> the latest version. Postgresql puts the unversioned packages in a 
> seperate source
> package. This allows several versions to coexist (but you can only build 
> against
> one) but at least the client and server tools could coexist. However the 
> more
> I thought about this (and talked to the release team a little), the less
> I can see that the Postgresql model is better.
> 

The pgsql model is cool, but I don't think mysql demands this
co-existing. Also the complexity added would also add burden to our
over-worked team as it is.

If people really want different versions of mysql, there's always
chroots/containers.

> c.) Another useful way thinking about the packages is "stacks". We have the
> following stacks:
>      server: mysql-server-core-N, mysql-server-N, mysql-server
>      client: mysql-client-core-N, mysql-client-N, mysql-client     test: 
> mysql-testsuite-core-N, mysql-testsuite-N, mysql-testsuite
>          [ only a semi-stack because it has to have the same version as 
> server ]
>      source: really on its own
>      library: libmysqlclientS, libmysqlclient-dev, libmysqld-dev & 
> libmysqld-pic
> A stack is a grouping of packages that can be thought of as a unit - and a
> way of breaking up the relationships between packages into more manageable
> chunks. So the dependencies between stacks should be as minimal and
> as loosely coupled as possible.
> So what should happen during a major version upgrade? Well mysql-common 
> should
> always be backwards compatible so that upgrade just goes through. The
> *unversioned* top of stack X tries to upgrade - but will have 
> dependencies on
> the new package X-N just below - and X-N will have Breaks/Replaces 
> clauses against
> X-P (where P<N). So the whole sequence is:
> i.) Version N of X gets downloaded and uppacked
> ii.) However it has an unfulfilled dependency on X-N so that gets downloaded
> and unpacked.
> iii.) X-N however has Breaks/Replaces against X-P so X-P gets deconfigured.
> iv.) X-N can now get configured and installed
> v.) X-P now gets removed
> vi.) Version N X can now get installed.
> 

This all makes sense but I'm not really sure what you're trying to get
at. This is all just standard package relationships.

> 4.) As noone has expressed any strong opinions I am proposing to split the
> mysql-server-N and mysql-client-N packages into the following:
>      innotop - separate source, mysql-client-N will depend on and later 
> recommend it
>      mysql-myisam-utils - those server side utilities associated with MyISAM
>      mysql-server-utils - non-essential server side utilities
>      text-replace - the replace program, needed by mysql-myisam-utils
> 

What actual issue are you trying to solve with the *-utils packages? I
don't see anybody complaining that myisamchk is included with
mysql-server-N.. and its basically never useful without a mysql server
to actually use the files with.

> 5.) I think it would be good to have the mysql packages sharing a docs 
> directory.
> It would save space and actually would be logical.  For technical
> reasons the base package has to be arch:any. mysql-common is not a good 
> choice
> actually because it contains the my.cnf and it is a good idea for 
> mysql-client-core-N, mysql-server-core-N and mysql-source-N not to have that
> automatically. So the proposal would be that:
> i.) The magic arch:any package containing the copyright, and upstream 
> and Debian
> changelogs would be called mysql-N (i.e. the same as the source package).
> ii.) mysql-common would be renamed mysql-config, but would still Provide
> mysql-common,  and would depend on mysql-N so it would look a lot like the
> current mysql-common. It would also contain a README.Debian explaining
> the reasoning behind the new structure.

So, 3x the complexity, for how much space savings?

> 
> 6.) Config modularity. The modern way would
> be for mysql-config to store a skeleton /etc/mysql/my.cnf and something 
> to keep
> /etc/mysql/conf.d alive. However we will have to keep just enough
> in /etc/mysql/my.cnf not to break the existing 5.5. Each of the packages
> would own its appropriate config fragment below /etc/mysql/conf.d .
> 

+1 for this, the old monolithic "one my.cnf to rule them all" way just
plain sucks. :)

> 7.) We use pkg-kde-tools to do the right thing by the shlibs system.
> 

Whats broken now again? Bad symbols being exported?

> 8.) I said that the dependencies between stacks should be minimized.
> i.) the client stack has dependencies on the library package because of
> the Perl programs. If the Perl scripts were in a seprate package that
> was only Recommended this dependency would be loosened.
> ii.) The server stack depends on the client stack because of check scripts
> that run during start up. With a bit of analysis this could probably be 
> tightened
> but at the very least I see no reason to require an exact match of minor 
> version
> across this interface.
> iii.) Obviously the test suite depends fundamentally and extensively on 
> the client
> and server (and maybe even library) stacks.


I see zero reason to relieve the servers from the client dependencies. 99%
of the time I have a functioning server, I want to login with full root
privileges locally at some point to look at stuff.

The stacks idea is fine, but you've provided no reason why the stacks
shouldn't be interdependent.

> 
> 9.) I have tried to put together a diagram of how I envisage the new package
> structure to look:
> http://www.periapt.co.uk/img/arcana/mysql.png
> 
> It tries to express visually what I mean by stacks.
> 
> I have a cunning plan
> ---------------------
> 1.) We talk about it. If noone discusses this stuff with me
> then I shall just go and implement it - albeit in experimental.

Please lets talk more first.

> 2.) One issue I know will get a discussion - how to acheive piuparts 
> compliance.
> I favour unconditional compliance. That means that, after every conceivable
> warning and opportunity to back up data and a NEWS item on upgrade, if the
> user says 'apt-get purge' (note purge - not  a mere 'remove') then 
> /var/lib/mysql
> gets nuked.  On the other hand Olaf has volunteered to talk to the 
> piuparts team
> about whether MySQL (and presumably other databases) get an exception. 
> However
> before he does that, I think we should guage how much consensus there is on
> the mailing list over the issue.

purge is known to delete data. We have a question, lets raise its priority
and flip the default. Note that on upgrade we should not change the value.
I'm not sure if debconf does this for us.

> 3.) Once there is a consensus over the package restructuring I would
> feel more comfortable about bringing Debian into line with Ubuntu on the 
> mysql-client-core-5.5 and mysql-testsuite changes.

Why wait? The changes are useful in Ubuntu, and the restructuring seems
like a whole lot of work.

> 4.) I will build a test framework to automate as much testing described in
> http://wiki.debian.org/Teams/MySQL/PreuploadTests  as possible. In 
> particular
> this means automated building of reverse dependencies.
> 5.) I implement the package restructuring described above (subject to the
> outcome of any discussions).
> 6.) Then based upon Olaf's discussions with the piuparts team,
> I will implement piuparts compliance and fix any bugs that are relevant.
> 7.) At this point we could play with MariaDB and Percona as drop in 
> replacements
> for mysql-server-core-N.
> 8.) By this point I expect to have played with apparmor in Debian to have
> backported the Ubuntu changes to unstable (as long as its post freeze). Of
> course I would have no issue with Clint doing so in the meantime.
> 9.) By this point I also expect to have played with upstart. However I never
> expect upstart to be default in Debian, and if the package is to be 100%
> synchronized between Ubuntu and Debian, then there will have to be 
> changes on the Ubuntu side. Also there are issues with upstart in Debian 
> currently.
> 10.) We use pkg-kde-tools and fix all the library issues. #673574 etc
> 11.) By this point I would expect the package to be lintian clean.
> 12.) By this point I would expect 5.6 to have been released. We would then
> give a heads up to the Debian release team.
> 13.) Then we would upload 5.6 to experimental and start raising bugs.
> 

Thanks for thinking this through Nicholas. I think you've signed up
for most of the work yourself, so its hard to say "why spend time on
that?" but I do think that change needs a good reason.. and I don't see
strong reasoning behind some of the changes proposed.