Page MenuHomePhorge
Create Task
Maniphest T2729

pykolab error/trouble with Umlaute - again
Closed, ResolvedPublic
Actions

Description

Kolab 16 on Debian 8

pykolab is still running into problems with Umlaute.

A similar problem with pykolab (or rather wallace) was fixed in 2015 ? https://issues.kolab.org/show_bug.cgi?id=4953
I also tried to post it to the list to find help or help fight the bug: http://lists.kolab.org/pipermail/users/2017-October/021305.html

I am pretty sure this error is crashing pykolab which in turn is responsible so that I can create new users but the mailbox wont be created.
The User with Umlaut ö is already existing since kolab 3.3

This error does not happen on a identical system with different users (no umlaute)

2017-11-17 15:17:27,205 pykolab.utils WARNING Could not translate Xöxxxxx.xxxxxx@xxxxxxxx.de using locale de_DE
2017-11-17 15:17:27,206 pykolab.auth ERROR An error occured using _paged_search: UnicodeEncodeError('ascii', u'(&(|(mail=x\xf6xxxxxx.xxxxxxx@xxxxxxxx.de)(alias=x\xf6xxxxxx.xxxxxxx@xxxxxxxxxx.de)(nsuniqueid=x\xf6xxxxx.xxxxxxx@xxxxxxxxxx.de))(!(nsuniqueid=27711d01-601d11e4-afec87db-51920d94)))', 11, 12, 'ordinal not in range(128)')
2017-11-17 15:17:27,206 pykolab.auth ERROR Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/pykolab/auth/ldap/__init__.py", line 3065, in _search
    secondary_domains
  File "<string>", line 10, in <module>
  File "/usr/lib/python2.7/dist-packages/pykolab/auth/ldap/__init__.py", line 2842, in _paged_search
    callback(entry=_result_data)
  File "/usr/lib/python2.7/dist-packages/pykolab/auth/ldap/__init__.py", line 2676, in _synchronize_callback
    eval("self._change_none_%s(entry, None)" % (entry['type']))
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/pykolab/auth/ldap/__init__.py", line 2186, in _change_none_user
    entry_changes = self.recipient_policy(entry)
  File "/usr/lib/python2.7/dist-packages/pykolab/auth/ldap/__init__.py", line 971, in recipient_policy
    results = self.find_recipient(_primary_mail, entry['id'])
  File "/usr/lib/python2.7/dist-packages/pykolab/auth/ldap/__init__.py", line 747, in find_recipient
    attrsonly=True
  File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 552, in search_s
    return self.search_ext_s(base,scope,filterstr,attrlist,attrsonly,None,None,timeout=self.timeout)
  File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 870, in search_ext_s
    return self._apply_method_s(SimpleLDAPObject.search_ext_s,*args,**kwargs)
  File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 812, in _apply_method_s
    return func(self,*args,**kwargs)
  File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 545, in search_ext_s
    msgid = self.search_ext(base,scope,filterstr,attrlist,attrsonly,serverctrls,clientctrls,timeout,sizelimit)
  File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 541, in search_ext
    timeout,sizelimit,
  File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 99, in _ldap_call
    result = func(*args,**kwargs)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf6' in position 11: ordinal not in range(128)

Details

Ticket Type
Task

Event Timeline

catwiesel created this task.Nov 17 2017, 4:05 PM
catwiesel updated the task description. (Show Details)
timeberhardt subscribed.Dec 7 2017, 9:29 AM

I can reproduce this bug with a fresh Kolab install on top of Ubuntu 16.04. I could not create new users after I added a user with an umlaut in his name.

The Bug is caused because in pykolab.utils the locale_name configured for the user (in my case de_DE) is fed to the python method locale.normalize which then returns de_DE.ISO8859-1. But the input string isn't latin but utf-8, so iconv can't convert the whole string and replaces the umlaut with a questionmark. As soon as I hardcoded utf-8 into the iconv call, everything was working again (of course that's not a solution).

catwiesel added a comment.Jan 9 2018, 7:01 PM
In T2729#48073, @timeberhardt wrote:

I can reproduce this bug with a fresh Kolab install on top of Ubuntu 16.04. I could not create new users after I added a user with an umlaut in his name.

The Bug is caused because in pykolab.utils the locale_name configured for the user (in my case de_DE) is fed to the python method locale.normalize which then returns de_DE.ISO8859-1. But the input string isn't latin but utf-8, so iconv can't convert the whole string and replaces the umlaut with a questionmark. As soon as I hardcoded utf-8 into the iconv call, everything was working again (of course that's not a solution).

Thank you for reproducing and confirming the problem.

Is the hardcoding workaround something I can do (text editor and your instructions) ?
I realise this is not a solution and I would very much prefer the bug be squashed and the fix be put into production, but as it stands, I really need to create new users/shared folders and would prefer a dirty solution over the right solution at an undetermined time.

feel free to mail me at my username here at gmx dot net

timeberhardt added a comment.Jan 11 2018, 2:22 PM

That's my dirty hack (utf-8 hardcoded):

--- /usr/lib/python2.7/dist-packages/pykolab/utils.py   2017-05-31 12:35:26.000000000 +0200
+++ utils.py    2018-01-11 14:21:58.956577689 +0100
@@ -543,8 +543,10 @@
                 '-t', 'ASCII//TRANSLIT',
                 '-s' ]
 
+    locale_charset = 'utf-8'
+    locale_string = '{0}.{1}'.format(locale_name, locale_charset)
     log.debug(_("Executing '%s | %s'") % (r"%s" % (mystring), ' '.join(command)), level=8)
-    process = subprocess.Popen(command, stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE, env={'LANG': locale.normalize(locale_name)})
+    process = subprocess.Popen(command, stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE, env={'LANG': locale_string})
 
     try:
         print >> process.stdin, r"%s" % mystring
catwiesel added a comment.Jan 24 2018, 1:42 PM
In T2729#49743, @timeberhardt wrote:

That's my dirty hack (utf-8 hardcoded):

...

Now that I have some time, I wanted to thank you. I patched utils.py and my problems were gone.

I am contemplating letting it run like that unless you can see a common situation in which your dirty hack will cause problems. I don't see any, but then again, your python expertise is far greater than mine... :)

mollekopf closed this task as Resolved.Aug 10 2023, 10:12 AM
mollekopf claimed this task.
mollekopf subscribed.

I think this got fixed in the python3 migration. Please reopen if you run into this issue again on a python3 system (>= Debian 11)