[Python-apps-team] Bug#682551: UnicodeDecodeError with some files in the file browser

[Antoine Beaupré]

Package: pykaraoke
Version: 0.7.5-1
Severity: minor

I get the following error when browsing my 'incoming' directory where
i threw a bunch of cdg/zip files.

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc9 in position 8: ordinal
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/pykaraoke.py", line 1670, in OnFileSel
    self.PopulateFolder(selected_node)
  File "/usr/lib/python2.7/dist-packages/pykaraoke.py", line 1602, in PopulateF
    if os.path.isdir (os.path.join (full_path, item)):
  File "/usr/lib/python2.7/posixpath.py", line 71, in join
    path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc9 in position 8: ordinal

I could the actual file to be:

item: Code d'?thique - la pyramide des affranchis (2003)

... by doing a print in that loop on line 1602. This filename is
indeed encoded weird, but that should be dealt with by the package,
and should crash the listing.

Here is the binary dump of the directory listing (sort of):

anarcat at marcos:incoming$ ls -d Code\ d\'* | hd
00000000  43 6f 64 65 20 64 27 c9  74 68 69 71 75 65 20 2d  |Code d'.thique -|
00000010  20 6c 61 20 70 79 72 61  6d 69 64 65 20 64 65 73  | la pyramide des|
00000020  20 61 66 66 72 61 6e 63  68 69 73 20 28 32 30 30  | affranchis (200|
00000030  33 29 0a                                          |3).|
00000033

I am not sure how to solve this, other than just skipping that nasty
file that python doesn't like. I tried a lot of combinations of
unicode(), encode() and decode() and gave up again. :P Here's a
workaround that skips the item:

--- /usr/lib/python2.7/dist-packages/pykaraoke.py.orig	2012-07-23 11:48:14.704823786 -0400
+++ /usr/lib/python2.7/dist-packages/pykaraoke.py	2012-07-23 11:48:56.093295766 -0400
@@ -1599,12 +1599,15 @@
         dir_list = []
         file_list = []
         for item in filedir_list:
-            if os.path.isdir (os.path.join (full_path, item)):
-                dir_list.append(item)
-            else:
-                root, ext = os.path.splitext(item)
-                if self.KaraokeMgr.SongDB.IsExtensionValid(ext):
-                    file_list.append(item)
+            try:
+                if os.path.isdir (os.path.join (full_path, item)):
+                    dir_list.append(item)
+                else:
+                    root, ext = os.path.splitext(item)
+                    if self.KaraokeMgr.SongDB.IsExtensionValid(ext):
+                        file_list.append(item)
+            except UnicodeDecodeError:
+                continue
         dir_list.sort()
         file_list.sort()
 
This looks nasty because of the indentation, but it's basically just:

--- /usr/lib/python2.7/dist-packages/pykaraoke.py.orig	2012-07-23 11:48:14.704823786 -0400
+++ /usr/lib/python2.7/dist-packages/pykaraoke.py	2012-07-23 11:48:56.093295766 -0400
@@ -1599,12 +1599,15 @@
         dir_list = []
         file_list = []
         for item in filedir_list:
+            try:
             if os.path.isdir (os.path.join (full_path, item)):
                 dir_list.append(item)
             else:
                 root, ext = os.path.splitext(item)
                 if self.KaraokeMgr.SongDB.IsExtensionValid(ext):
                     file_list.append(item)
+            except UnicodeDecodeError:
+                continue
         dir_list.sort()
         file_list.sort()
 
Hopefully that will be useful for someone out there. :)

-- System Information:
Debian Release: wheezy/sid
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 3.2.0-2-amd64 (SMP w/2 CPU cores)
Locale: LANG=fr_CA.UTF-8, LC_CTYPE=fr_CA.UTF-8 (charmap=UTF-8) (ignored: LC_ALL set to fr_CA.UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages pykaraoke depends on:
ii  pykaraoke-bin     0.7.5-1
ii  python            2.7.3~rc2-1
ii  python-pygame     1.9.1release+dfsg-6
ii  python-pykaraoke  0.7.5-1
ii  python-wxgtk2.8   2.8.12.1-11

pykaraoke recommends no packages.

Versions of packages pykaraoke suggests:
ii  python-numpy  1:1.6.2-1

-- no debconf information