[Python-modules-team] Bug#611923: python-xdg: wrong behaviour breaking apps on non-supported locale

[Jakub Wilk]

Disclaimer: I'm not maintainer of this package.

* Yann Dirson <ydirson at free.fr>, 2011-02-03, 20:56:
>In [1]: import xdg.DesktopEntry
>
>In [3]: e=xdg.DesktopEntry.DesktopEntry()
>
>In [4]: e.parse('plugins/Games/Chess.desktop')
>
>In [5]: e.getName()
>Out[5]: u'\xc9checs'
>
>In [6]: print "%s" % e.getName()
>------> print("%s" % e.getName())
>Échecs
>
>
>Now, if I use LC_ALL=fr_FR or fr_FR.ISO-8859-1 (which should be
>equivalent), the final step instead throws:
>
>UnicodeEncodeError: 'ascii' codec can't encode character u'\xc9' in position 0: ordinal not in range(128)

I assume that, as the subject suggest, it fails only if there is no 
fr_FR.ISO-8859-1 locale available. Am I correct? (If this is the case, 
perhaps something like fr_FR.ISO-8859-42 would be a better test-case, as 
it's less like to exist.)

>By contrast, other programs in the same condition fallback to C
>locale, which results in no error.  I guess the xdg module should do
>something similar.

It is true that xdg uses a bit different language lookup algorithm that 
GNU gettext does. I can see it is flawed in a few ways and I can see 
your point. However, I don't think your particular use case is of much 
significance, for the following reasons:

1. If you are using non-existent locales, you shoot yourself in the 
foot. :) 

2. getName() returned a Unicode string for in French, which was kind of 
what you asked for.

3. If you want your application to be robust, you should not print 
Unicode strings blindly. Encoding of sys.stdout can be ASCII even if 
proper UTF-8 locale is set:

$ locale charmap
UTF-8

$ python -c 'print u"\xde"'
Þ

$ python -c 'print u"\xde"' | cat
Traceback (most recent call last):
   File "<string>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xde' in position 0: ordinal not in range(128)

-- 
Jakub Wilk