[Openstack-devel] Rewriting the .ini parsing bit of openstack-pkg-tools

[Thomas Goirand]

Hi,

This often happens when one do some code: you hack, hack and hack
around, then after a while, you realize your code need some clean up.
I'm at this point in time with the ini parsing functions in all core
packages of Openstack.

As we're supposed to be a team (even though I really feel too much alone
doing the heavy work...), I'd like to discuss the problem publicly.

When Roland was doing some packaging work, we had discussions on how to
do the parsing of .ini files inside Openstack. I first wrote a very ugly
shell script function, which was about 150 lines, which is completely
lame, for such a small task. Then he decided (rightly) that it was crap,
and tried rewriting it in python. The current code in
openstack-pkg-tools is from him, and is like this:

python -c "import configobj
config=configobj.ConfigObj('${FILE}')
config['${SECTION}']['${DIRECTIVE}']='${VALUE}'
config.write()"

Unfortunately, the Openstack .ini files aren't really ini files, and can
have entries this way:

[composite:ec2]
use = egg:Paste#urlmap
/services/Cloud: ec2cloud

As a consequence, the class configobj which is in the minimal default
python installation in Debian crashes. Roland then decided to patch the
nova api-paste.ini to fix the problem, and transform some of the
directives this way:

diff -u etc/nova/api-paste.ini.orig etc/nova/api-paste.ini:
[...]
-/: meta
+/ = meta
[...]

This somehow worked, but unfortunately, what none of us saw is that the
Essex version of some Openstack packages, currently in Wheezy, does
contain some non-standard .ini files. As a consequence, upgrade crashes,
as per this bug:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=700620

More over, in Openstack Grizzly, which will be out soon, there's even
more URLs like above, so we have even more problems.

So, I'm at a point where the python code using configobj is not a viable
solution. I didn't like using it and patching nova api-paste.ini (I'm
not trying to put the blame on Roland, but it wasn't my idea), but now
I'm even more convince it's not the solution to our problem.

In Grizzly, there's a new python module called python-oslo.config. It's
very easy to use, and knows how to parse. The problem is that using a
python module means adding a Pre-Depends, which breaks the flow of
Debconf. Indeed, here's what it does currently:

# apt-get install openstack-proxy-node
1/ some downloads
2/ Absolutely all debconf questions
3/ Installation of all packages

But if I add a Pre-Depends: python-oslo.config, it will go like this:

# apt-get install openstack-proxy-node
1/ some downloads
2/ some debconf questions, for example mysqld password
3/ some install with dpkg, including python-oslo.config
4/ some more questions for the packages that have the Pre-Depends:
5/ some install with dpkg of the packages not finished to install

I don't want the 2nd version. Openstack operator all want to answer all
questions, then go to the coffee machine and rest 10 minutes during the
setup... ;)

So, I continue to think that everything should be done in shell script.
Not the way I wrote it the first time though. The goal is to have the
same implementation as this:
https://github.com/openstack/oslo-config/blob/master/oslo/config/iniparser.py

So, I wrote 2 versions: one in pure sh, and one using awk. Both are
using regular expression, so they are easy to fix if the config files
where to evolve in format (which is most probable seeing the history of
nova.conf: from flag file, to config file, to .ini format, to what it is
now ...).

I have attached both versions, and I would like my you, my fellow
Openstack Debian packaging team members, to comment on the
implementation. Which of the 2 seems the most maintainable, easy to
understand, and above all the most correct and bug free?

* Note that I'm aware that the awk version doesn't do all of what is
described before the function, and would need some more work. *

Would any of you use a totally different approach to the problem?

For the moment, I would lean toward the grep + sed version, as I see it
easier to read and understand, plus there's a unique parsing logic for
both get and set, which is nicer.

If you have suggestions on how to improve, or do it better in a totally
different way, please do explain how you would do it and share some code.

*** WARNING ***
With these functions, I'm breaking the current openstack-pkg-os API, so
I will have to edit all core projects (nova, cinder, keystone, glance,
quantum, ceilometer and heat). I do *expect some breakage* doing it,
since I'm also changing parameter order calling, and removing some
parameters which I think are bad. All this is error prone. So if you
need to get a working snapshot of Folsom from my FTP, please do it right
now, before I start uploading new versions. As for the Grizzly release
G3, it's currently broken anyway until all this work is done (so please
be patient and wait for me to fix cinder, quantum and nova in Grizzly,
who are failing at the postinst stage currently).

Thanks already if you read all the above, and even more for all those
who will take the time to voice an opinion and contribute a better
implementation.

Cheers,

Thomas
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: parse_ini_awk
URL: <http://lists.alioth.debian.org/pipermail/openstack-devel/attachments/20130311/9c2609b9/attachment-0006.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: parse_ini_sh
URL: <http://lists.alioth.debian.org/pipermail/openstack-devel/attachments/20130311/9c2609b9/attachment-0007.ksh>