[debian-mysql] Groping for a roadmap in the dark

[Nicholas Bamber]

On 08/06/12 06:39, Clint Byrum wrote:
> Excerpts from Nicholas Bamber's message of 2012-06-03 15:48:39 -0700:
>> There are currently three major differences between Ubuntu and Debian's
>> repository:
>> 1.) Ubuntu uses upstart rather than init.
>> upstart looks like great and I would like Debian to support it. However
>> currently
>> it is not available in on non-Linux and I doubt it will ever be the default.
>> Therefore Debian must find a way of working with upstart if it is in use
>> and using traditional init otherwise. If we can achieve that I do not see
>> why ubuntu should require any customizations but that would preclude
>> mysql-server-5.5 depending on upstart as it does in ubuntu.
>
> There is a policy change needed to make upstart viable in Debian. If you
> feel that upstart should be in Debian (I know I do) then please go here
> and voice your opinion:
>
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=591791

The policy change seems inocuous enough. However I am so going to keep 
my head down on that one. I am sure Debian will get there eventually.

>
>> 2.) I have no issue with the apparmor stuff in principle. I am still trying
>> to work my way through the documentation. However until the other two issues
>> look more tractable I fail to see resolving this as  a high priority.
>
> The delta for Ubuntu to have this is pretty tiny, so I don't think its
> too important to worry about this for that reason. As AppArmor gets
> better in Debian, we should find it easy to turn it on though. Lets
> shelve this until other things are worked out.

Well if you are happy with that then okay. I am having enough trouble 
with the things I think I understand let alone the ones I know I don't.

>
>> 3.) The packet restructuring (mysql-client-core-5.5, mysql-testsuite) -
>> The rest of this email is really about that.
>>
>> Principles of Good Package Structure
>> ------------------------------------
>> I cannot profess to understand good package structure but I am pretty sure
>> we don't have it. So I am groping to find what that would be.
>>
>> 1.) Piuparts is a statement of how a Debian package should behave.
>> Essentially
>> a package should not stomp over its own or other packages and should
>> clean up
>> after itself when asked (i.e. when asked to do a purge). We simply don't
>> comply at the moment. Given Debian MySQL's bad history on this we will
>> have to take extra care but I feel it has to be done.
>>
>
> Piuparts is an awesome assessment of where we stand w.r.t. policy. Its true,
> we are not automatically purging data if the question about whether or not
> to remove /var/lib/mysql/* is left to defaults.
>
> I'd entertain making mysql-server-5.5/postrm_remove_databases default to true
> rather than false, as long as it is noted in NEWS, and the question is given
> a high priority. As I understand it, this would solve the issue entirely.

It seems we are in agreement. Basically I think we need to do everything 
we can to alert the user to this change. This will include having the 
change in either experimental or unstable for most of wheezy+1.

Olaf however will doubtless disagree. I would like to make sure he has 
an opportunity to make his case. There is a while before I get  to 
piuparts on the roadmap.

>
>> 2.) The client/server divide. There should be a clear divide between the two
>> (apart from mysql-common). They should be happy to coexist on the same
>> machine
>> or be on separate machines and it should be possible to upgrade one and
>> not the
>> other.  A client side executable is one that only needs
>> a network connection to communicate with the server. A server side
>> machine needs
>> to be on the same machine. Again the current package does not conform.
>
> All good ideas, but I don't know that we need to do anything to change
> this.  What specifically is overlapping here?

I made notes on all restructuring issues at 
http://wiki.debian.org/Teams/MySQL/WhichPackage. But here are the bits 
relevant to this point.

a.) innochecksum (a server utility by my definition) has the man page on 
the server side and the executable on the client side.
b.) myisam_ftdump, mysqldumpslow, mysql_find_rows, mysql_fix_extensions, 
mysql_plugin are currently on the client side but by my definition 
should be on the server side.
c.) mysql_convert_table_format, mysql_setpermission are currently on the 
server side but by my definition should be on the client side.

>
>>
>> 3.) A smooth upgrade path. We had issues this time. I am trying to
>> understand what the theory is.
>>
>> a.) mysql-common should always be backwards compatible (not indefinitely
>> but for
>> for the most recent previous major version). This can be achieved by
>> bringing in
>> new options as "loose" options. When we move them to non-loose we put in
>> Breaks
>> clauses the
>> old package. This would allow, say, the client side to be upgraded
>> whilst the server side was kept back. The way it was done this time was
>> defensible but went
>> because of other issues. To adopt this policy would be regarded as defensive
>> package management.
>
> Yeah I didn't know we had the option of the 'loose' options, this will
> allow us to more smoothly transition on major version updates. This
> seems like a no-brainer, lets do that.


Great

>
>>
>> b.) I think the packages can be usefully divided into three "slices":
>> unversioned,
>> versioned and build-time. The versioned packages (libmysqlclientS,
>> mysql-server-core-N, etc) are used in run-time. The unversioned packages
>> are very light or virtual packages that provide some naming stability
>> across major versions.
>> Build-time packages being unversioned means you can only build against
>> the latest version. Postgresql puts the unversioned packages in a
>> seperate source
>> package. This allows several versions to coexist (but you can only build
>> against
>> one) but at least the client and server tools could coexist. However the
>> more
>> I thought about this (and talked to the release team a little), the less
>> I can see that the Postgresql model is better.
>>
>
> The pgsql model is cool, but I don't think mysql demands this
> co-existing. Also the complexity added would also add burden to our
> over-worked team as it is.
>
> If people really want different versions of mysql, there's always
> chroots/containers.

I spent a long time thinking about the postgresql model (and talking to 
the release team - and getting an RC bug for my pains ;-) ). At the end 
of it I came to the conclusion that the postgresql model has nothing to 
offer. Your idea of just having an unversioned source package is fine.


>
>> c.) Another useful way thinking about the packages is "stacks". We have the
>> following stacks:
>>       server: mysql-server-core-N, mysql-server-N, mysql-server
>>       client: mysql-client-core-N, mysql-client-N, mysql-client     test:
>> mysql-testsuite-core-N, mysql-testsuite-N, mysql-testsuite
>>           [ only a semi-stack because it has to have the same version as
>> server ]
>>       source: really on its own
>>       library: libmysqlclientS, libmysqlclient-dev, libmysqld-dev&
>> libmysqld-pic
>> A stack is a grouping of packages that can be thought of as a unit - and a
>> way of breaking up the relationships between packages into more manageable
>> chunks. So the dependencies between stacks should be as minimal and
>> as loosely coupled as possible.
>> So what should happen during a major version upgrade? Well mysql-common
>> should
>> always be backwards compatible so that upgrade just goes through. The
>> *unversioned* top of stack X tries to upgrade - but will have
>> dependencies on
>> the new package X-N just below - and X-N will have Breaks/Replaces
>> clauses against
>> X-P (where P<N). So the whole sequence is:
>> i.) Version N of X gets downloaded and uppacked
>> ii.) However it has an unfulfilled dependency on X-N so that gets downloaded
>> and unpacked.
>> iii.) X-N however has Breaks/Replaces against X-P so X-P gets deconfigured.
>> iv.) X-N can now get configured and installed
>> v.) X-P now gets removed
>> vi.) Version N X can now get installed.
>>
>
> This all makes sense but I'm not really sure what you're trying to get
> at. This is all just standard package relationships.

Yes. As with postgresql above this my struggling with making sure I 
understand how these complex packages work.

>
>> 4.) As noone has expressed any strong opinions I am proposing to split the
>> mysql-server-N and mysql-client-N packages into the following:
>>       innotop - separate source, mysql-client-N will depend on and later
>> recommend it
>>       mysql-myisam-utils - those server side utilities associated with MyISAM
>>       mysql-server-utils - non-essential server side utilities
>>       text-replace - the replace program, needed by mysql-myisam-utils
>>
>
> What actual issue are you trying to solve with the *-utils packages? I
> don't see anybody complaining that myisamchk is included with
> mysql-server-N.. and its basically never useful without a mysql server
> to actually use the files with.

Well there are three bug reports asking for splitting: #660087 
(innotop), #317934  (replace), #511438 (vague rant about the size of the 
server side). And you have accommodated requests for splitting before 
with client-core and I had to ask you what the reason was as there was 
nothing clear in the changelog.

I am not really a splitter or a lumper. It's a matter of preference and 
one I do not feel strongly about. Each case is different and needs to be 
  treated on its merits. However we ought to have agreed and documented 
principles. It is flushing those out that I am really about.

innotop seems the most clear cut. It's a separate source and upstream is 
Google. We make our lives easier by splitting it out.

replace is the next most clear cut. The  user claims it is useful in its 
own right. You seem to have been sympathetic to such claims in the past 
with the core packages. I guess we could go back to the user and ask 
them to explain how they would use the "replace" utility. But short of 
that I would predict you agreeeing to that split.

The real debate I think is around #511438. I think there is no issue 
with moving things only used by the test stack into the test stack apart 
from identifying them correctly. Marc Lehmann is wrong about resolveip 
as it used in the installation scripts. I have no sympathy with him on 
resolve_stack_dump and perror. They may be rarely used but they are for 
when things go wrong and I think ought to be available. But his (and 
Yauhen Kharuzhy's) general point that the mysql-server (also including 
mysql-client and libmysqlclient18) is large is I think reasonable.

The only way I can see that we could address that concern is by asking 
which bits of the server and client stacks are *optional*?

* Well a highly transactional database is unlikely to have MyISAM 
tables. Then all the MyISAM utilities would be unneeded.
* Some databases optimized for speed might have only MyISAM tables. Such 
a setup would have no need for say innochecksum and possibly others.
* Executables that rely on the DBD::MySQL Perl module could be made 
optional. This would break the mandatory dependency between the library 
stack and the other stacks. On the client side this would affect 
mysql_convert_table_format, mysqlreport, mysq_setpermission. On the 
server side this would impact mysqlhotcopy which is only relevant to 
MyISAM tables.

I am also slightly surprised you did not choke on my idea of splitting 
mysql-testsuite-5.5 into arch:all and arch:any bits. I don't think in 
practice the arch:any bit would be of much use. The argument is based 
entirely on it being a waste to have an arch:any package consisting 
almost entirely of text files.

So I throw the question back to you. How would you respond to #511438?

So what I want is a clear set of principles  that will allow us to:
1.) Know where to place any new files.
2.) Know how to respond promptly and authoritatively to requests for 
splitting.
3.) avoid file clashes in the future.
4.) In accordance with those principles head off future requests by 
dealing preemptively with any at the beginning of the coming release cycle.


>
>> 5.) I think it would be good to have the mysql packages sharing a docs
>> directory.
>> It would save space and actually would be logical.  For technical
>> reasons the base package has to be arch:any. mysql-common is not a good
>> choice
>> actually because it contains the my.cnf and it is a good idea for
>> mysql-client-core-N, mysql-server-core-N and mysql-source-N not to have that
>> automatically. So the proposal would be that:
>> i.) The magic arch:any package containing the copyright, and upstream
>> and Debian
>> changelogs would be called mysql-N (i.e. the same as the source package).
>> ii.) mysql-common would be renamed mysql-config, but would still Provide
>> mysql-common,  and would depend on mysql-N so it would look a lot like the
>> current mysql-common. It would also contain a README.Debian explaining
>> the reasoning behind the new structure.
>
> So, 3x the complexity, for how much space savings?

Actually no I think this is natural. The "mysql" base package (which 
actually should not be versioned)  contains the copyright and changelogs 
and an explanation of how the package fits together. Everything needs 
this. It will be good to have one package that shares a name with the 
source package. It will actually make it easier for people to find their 
way round the docs directories.

The other half of the  old mysql-common would be mysql-config which 
would be exactly that. Not all the packages need that and so 
"mysql-common" is actually a bad name because the *core-5.5 packages 
don't need it.

>
>>
>> 6.) Config modularity. The modern way would
>> be for mysql-config to store a skeleton /etc/mysql/my.cnf and something
>> to keep
>> /etc/mysql/conf.d alive. However we will have to keep just enough
>> in /etc/mysql/my.cnf not to break the existing 5.5. Each of the packages
>> would own its appropriate config fragment below /etc/mysql/conf.d .
>>
>
> +1 for this, the old monolithic "one my.cnf to rule them all" way just
> plain sucks. :)


great

>
>> 7.) We use pkg-kde-tools to do the right thing by the shlibs system.
>>
>
> Whats broken now again? Bad symbols being exported?

Well we have lintian reports on that. There is a web page at 
http://wiki.debian.org/Projects/ImprovedDpkgShlibdeps . I see this as 
part of the wider question - are we doing everything we can to serve the 
users of our libraries? I would like to defer discussion of this but I 
do want to come back to it and I want to have a clear policy and 
documented procedures by the time of the next release.

>
>> 8.) I said that the dependencies between stacks should be minimized.
>> i.) the client stack has dependencies on the library package because of
>> the Perl programs. If the Perl scripts were in a seprate package that
>> was only Recommended this dependency would be loosened.
>> ii.) The server stack depends on the client stack because of check scripts
>> that run during start up. With a bit of analysis this could probably be
>> tightened
>> but at the very least I see no reason to require an exact match of minor
>> version
>> across this interface.
>> iii.) Obviously the test suite depends fundamentally and extensively on
>> the client
>> and server (and maybe even library) stacks.
>
>
> I see zero reason to relieve the servers from the client dependencies. 99%
> of the time I have a functioning server, I want to login with full root
> privileges locally at some point to look at stuff.
>
> The stacks idea is fine, but you've provided no reason why the stacks
> shouldn't be interdependent.

Well as you say there are dependencies and they are necessary. But for 
example mysql-server-5.5 depends on mysql-client-5.5 (>= 
${source:Version}). Clearly there will be times when a specific minimum 
version is required but surely this is not necessary and upgrades would 
be easier without this tight coupling.


>
>>
>> 9.) I have tried to put together a diagram of how I envisage the new package
>> structure to look:
>> http://www.periapt.co.uk/img/arcana/mysql.png
>>
>> It tries to express visually what I mean by stacks.
>>
>> I have a cunning plan
>> ---------------------
>> 1.) We talk about it. If noone discusses this stuff with me
>> then I shall just go and implement it - albeit in experimental.
>
> Please lets talk more first.

yes

>
>> 2.) One issue I know will get a discussion - how to acheive piuparts
>> compliance.
>> I favour unconditional compliance. That means that, after every conceivable
>> warning and opportunity to back up data and a NEWS item on upgrade, if the
>> user says 'apt-get purge' (note purge - not  a mere 'remove') then
>> /var/lib/mysql
>> gets nuked.  On the other hand Olaf has volunteered to talk to the
>> piuparts team
>> about whether MySQL (and presumably other databases) get an exception.
>> However
>> before he does that, I think we should guage how much consensus there is on
>> the mailing list over the issue.
>
> purge is known to delete data. We have a question, lets raise its priority
> and flip the default. Note that on upgrade we should not change the value.
> I'm not sure if debconf does this for us.

yes I think debconf will respect the old value. We should test that.

>
>> 3.) Once there is a consensus over the package restructuring I would
>> feel more comfortable about bringing Debian into line with Ubuntu on the
>> mysql-client-core-5.5 and mysql-testsuite changes.
>
> Why wait? The changes are useful in Ubuntu, and the restructuring seems
> like a whole lot of work.

Yes I am happy to make those changes now.

>
>> 4.) I will build a test framework to automate as much testing described in
>> http://wiki.debian.org/Teams/MySQL/PreuploadTests  as possible. In
>> particular
>> this means automated building of reverse dependencies.
>> 5.) I implement the package restructuring described above (subject to the
>> outcome of any discussions).
>> 6.) Then based upon Olaf's discussions with the piuparts team,
>> I will implement piuparts compliance and fix any bugs that are relevant.
>> 7.) At this point we could play with MariaDB and Percona as drop in
>> replacements
>> for mysql-server-core-N.
>> 8.) By this point I expect to have played with apparmor in Debian to have
>> backported the Ubuntu changes to unstable (as long as its post freeze). Of
>> course I would have no issue with Clint doing so in the meantime.
>> 9.) By this point I also expect to have played with upstart. However I never
>> expect upstart to be default in Debian, and if the package is to be 100%
>> synchronized between Ubuntu and Debian, then there will have to be
>> changes on the Ubuntu side. Also there are issues with upstart in Debian
>> currently.
>> 10.) We use pkg-kde-tools and fix all the library issues. #673574 etc
>> 11.) By this point I would expect the package to be lintian clean.
>> 12.) By this point I would expect 5.6 to have been released. We would then
>> give a heads up to the Debian release team.
>> 13.) Then we would upload 5.6 to experimental and start raising bugs.
>>
>
> Thanks for thinking this through Nicholas. I think you've signed up
> for most of the work yourself, so its hard to say "why spend time on
> that?" but I do think that change needs a good reason.. and I don't see
> strong reasoning behind some of the changes proposed.

What I am  really looking for is a bonfire of the bug reports. Let's 
have clear policies and tools and practices that keep us out of trouble 
and provide solid well thought out answers to wishlist bug reports that 
would otherwise just be a matter of opinion. The time to start on this 
is at the beginning of a release cycle and the time to plan it is now.

Another small issue  that belongs in this cycle is character sets: 
#321505 and #487461.

>
> _______________________________________________
> pkg-mysql-maint mailing list
> pkg-mysql-maint at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-mysql-maint