Page MenuHomePhorge

wallace does not deliver 8bit encoded utf-8 mails, throws UnicodeEncodeError
Closed, ResolvedPublic

Description

System: AlmaLinux release 9.2
Relevant rpm-packages:

python3-3.9.16-1.el9_2.1.x86_64
python3-libs-3.9.16-1.el9_2.1.x86_64
kolab-16.0.1-50.2.el9.kolab_16.x86_64
pykolab-0.9.0-10.2.el9.kolab_16.noarch
pykolab-xml-0.9.0-10.2.el9.kolab_16.noarch
wallace-0.9.0-10.2.el9.kolab_16.noarch

8bit encoded utf-8 mails are held back in /var/spool/pykolab/wallace/invitationpolicy/incoming and no bounces or delayed delivery notifications are send
the pylog shows

2023-07-06 14:37:29,901 pykolab.wallace/modules ERROR [14221] tmpp4vq8x3o smtplib - Unknown error occurred: UnicodeEncodeError('ascii', 'X-Virus-Scanned: amavis at la-bw.de\r\nX-Spam-Flag: NO\r\nX-Spam-Score: -0.799\r\nX-Spam-Level: \r\nX-Spam-Status: No, score=-0.799 tagged_above=-10 required=6.2\r\n tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1,\r\n RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001,\r\n TVD_SPACE_RATIO=0.001] autolearn=ham autolearn_force=no\r\nAuthentication-Results: kolab-test.la-bw.de (amavis); dkim=pass (2048-bit key)\r\n header.d=uni-tuebingen.de\r\nReceived: from mx03.uni-tuebingen.de (mx03.uni-tuebingen.de [134.2.5.213])\r\n\tby kolab-test.la-bw.de (Postfix) with ESMTPS id 2E0BE2083313\r\n\tfor <michael.menge@kolab-test.la-bw.de>; Thu, 29 Jun 2023 12:29:06 +0200 (CEST)\r\nReceived: from webmail2.uni-tuebingen.de (webmail2.uni-tuebingen.de [134.2.5.195])\r\n\tby mx03.uni-tuebingen.de (Postfix) with ESMTPS id 8233A20C5ECB\r\n\tfor <michael.menge@kolab-test.la-bw.de>; Thu, 29 Jun 2023 12:29:06 +0200 (CEST)\r\nDKIM-Filter: OpenDKIM Filter v2.11.0 mx03.uni-tuebingen.de 8233A20C5ECB\r\nDKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uni-tuebingen.de;\r\n\ts=20211202prod; t=1688034546;\r\n\tbh=ThiDwRAn565i8l6j5DBx01nxYUx/SAWpj/oSVJVbsBQ=;\r\n\th=Date:From:To:Subject:From;\r\n\tb=gVlv1mLJBFUrQkAFIJMkt6pUeP2wBNViomGY1MxP1ELL89ayEW5URVrzi2OcKMnHL\r\n\t Mo90CqloroOQd0E24e7UpXvedbzh8zaxuJlZdmYeY3cle19mQ2mQ/1G3MdhCy/bHBh\r\n\t WLbM6FXBJ02+DpUPRyWc2OgOMmwxgU0NGiDmC0Kbgdkyc/eG2dGOLhmk95OQzz76ur\r\n\t AucEv8o78qkrgckeZ0ucwNd+04BdoEJqUmrxfNr4nyRleIOEOe7mHsXRVN/XrB49hD\r\n\t hMkUrXwKKby3BHpk2OH8rxCpiCbtzZ6AZTU7lKo/kGMPx3M5BhY63hsswmeSglSm4n\r\n\t cQl4SVFqp4MWw==\r\nReceived: from maxwell.wi113.uni-tuebingen.de\r\n (maxwell.wi113.uni-tuebingen.de [134.2.232.77]) by webmail.uni-tuebingen.de\r\n (Horde Framework) with HTTPS; Thu, 29 Jun 2023 12:29:06 +0200\r\nDate: Thu, 29 Jun 2023 12:29:06 +0200\r\nMessage-ID: <20230629122906.Horde.49wBz0QiWviHsQ00Fr9Ok9V@webmail.uni-tuebingen.de>\r\nFrom: Michael Menge <michael.menge@zdv.uni-tuebingen.de>\r\nTo: michael.menge@kolab-test.la-bw.de\r\nSubject: test3\r\nUser-Agent: Horde Application Framework 5\r\nContent-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes\r\nMIME-Version: 1.0\r\nContent-Disposition: inline\r\nContent-Transfer-Encoding: 8bit\r\n\r\n\r\ntest\r\n\r\n--------------------------------------------------------------------------------\r\nMichael Menge                          Tel.: (49) 7071 / 29-70316\r\nUniversität Tübingen                   Fax.: (49) 7071 / 29-5912\r\nZentrum für Datenverarbeitung          mail:  \r\nmichael.menge@zdv.uni-tuebingen.de\r\nWächterstraße 76\r\n72074 Tübingen\r\n', 2318, 2319, 'ordinal not in range(128)')
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/wallace/modules.py", line 163, in _sendmail
    _response = smtp.sendmail(sender, recipients, msg)
  File "/usr/lib64/python3.9/smtplib.py", line 875, in sendmail
    msg = _fix_eols(msg).encode('ascii')
UnicodeEncodeError: 'ascii' codec can't encode character '\xe4' in position 2318: ordinal not in range(128)

The Mail file does not contain a "\xe4" (ä in iso-8859-1) character but a valid encoded utf-8 ä "c3a4"

[root@kolab-test ~]# file /var/spool/pykolab/wallace/invitationpolicy/incoming/tmpp4vq8x3o
/var/spool/pykolab/wallace/invitationpolicy/incoming/tmpp4vq8x3o: UTF-8 Unicode text, with CRLF, LF line terminators

[root@kolab-test ~]# hexdump -C /var/spool/pykolab/wallace/invitationpolicy/incoming/tmpp4vq8x3o | tail -n 19
*
000008e0  2d 2d 2d 2d 2d 2d 2d 2d  2d 0a 4d 69 63 68 61 65  |---------.Michae|
000008f0  6c 20 4d 65 6e 67 65 20  20 20 20 20 20 20 20 20  |l Menge         |
00000900  20 20 20 20 20 20 20 20  20 20 20 20 20 20 20 20  |                |
00000910  20 54 65 6c 2e 3a 20 28  34 39 29 20 37 30 37 31  | Tel.: (49) 7071|
00000920  20 2f 20 32 39 2d 37 30  33 31 36 0a 55 6e 69 76  | / 29-70316.Univ|
00000930  65 72 73 69 74 c3 a4 74  20 54 c3 bc 62 69 6e 67  |ersit..t T..bing|
00000940  65 6e 20 20 20 20 20 20  20 20 20 20 20 20 20 20  |en              |
00000950  20 20 20 20 20 46 61 78  2e 3a 20 28 34 39 29 20  |     Fax.: (49) |
00000960  37 30 37 31 20 2f 20 32  39 2d 35 39 31 32 0a 5a  |7071 / 29-5912.Z|
00000970  65 6e 74 72 75 6d 20 66  c3 bc 72 20 44 61 74 65  |entrum f..r Date|
00000980  6e 76 65 72 61 72 62 65  69 74 75 6e 67 20 20 20  |nverarbeitung   |
00000990  20 20 20 20 20 20 20 6d  61 69 6c 3a 20 20 0a 6d  |       mail:  .m|
000009a0  69 63 68 61 65 6c 2e 6d  65 6e 67 65 40 7a 64 76  |ichael.menge@zdv|
000009b0  2e 75 6e 69 2d 74 75 65  62 69 6e 67 65 6e 2e 64  |.uni-tuebingen.d|
000009c0  65 0a 57 c3 a4 63 68 74  65 72 73 74 72 61 c3 9f  |e.W..chterstra..|
000009d0  65 20 37 36 0a 37 32 30  37 34 20 54 c3 bc 62 69  |e 76.72074 T..bi|
000009e0  6e 67 65 6e 0a                                    |ngen.|
000009e5

Details

Ticket Type
Task

Event Timeline

quoted-printable encoded utf-8 mails are delivered without a problem

machniak subscribed.

Confirmed. The following patch seems to help even though I don't see BODY=8BITMIME to be used. So, it might be an incomplete solution.

--- a/wallace/modules.py
+++ b/wallace/modules.py
@@ -153,6 +153,10 @@ def _sendmail(sender, recipients, msg):
     if conf.debuglevel > 8:
         smtp.set_debuglevel(1)
 
+    # Workaround smtplib's issue with 8bit encoded messages
+    if isinstance(msg, str):
+        msg = msg.encode('utf-8')
+
     # Sender can't be a list
     if isinstance(sender, list):
         sender = sender[0]

Looking at smtplib code it just makes it skip the faulty _fix_eols() function.

https://github.com/python/cpython/blob/main/Lib/smtplib.py#L863

machniak lowered the priority of this task from Needs Triage to High.Jul 27 2023, 4:08 PM

I did apply the workaround patch and it did resolve the problem for my test case,
but i have not tested any other 8BITMIME Mails with non-UTF-8 charsets e.g. "ISO-8859-15" or "UTF-16"