Custom logger adapter for pykolab

Logger names for rest wallace modules

Make Wallace log message queue id. That makes it easier to track log records associated with a processed message. Queue ID is a file name of Wallace message

• fix error message by showing which template file was not found
• Move find_template function to utils as there is a plan to use it elsewhere.
• Changes to setup-kolab mta utility to support AD. Postfix LDAP lookup tables are now created using Jinja2 templates.
• Log clamav updates to /var/log/freshclam.log file. Don't want to create a dir.
• Jinja2 templates for Postfix and Amavisd configuration.

• Don't need this log.
• Fail immediatly if connection to LDAP server could not be made
• Get supported ldap controls and match with what is supported by pykolab

• Use python UUID to convert binary objectGUID to string. It doesn't look like value of uniqueid returned by from cache is used anywhere (needs to double check), so only forward conversion is needed.

• Use python UUID to convert binary objectGUID to string. It doesn't look like value of uniqueid returned by from cache is used anywhere (needs to double check), so only forward conversion is needed.
• Also do not strip anything from bytestring attributes (objectGUID) because after stripping conversion to UUID string is not possible.

Kind of finished version of Pykolab way of Active Directory setup:

• For AD LDAP modification I decided to use Jinja2 templates, which are in templates folder. Several reasons for that:
  • Long texts of changes to be performed in python code does not look nice, especially when AD Kolab schema needs to be added.
  • Ansible uses Jinja2 for templates, so templates used by setup_ldap.py can be reused with Ansible.
  • It becomes easier to load additional data to LDAP (like data to for testing purposes) or setup OpenLDAP for use with Kolab.

Using Jinja2 creates python-jinja2 package dependency
TODO - Need to decide where Jinja2 templates should be installed. Currently /etc/kolab/templates/, /usr/share/kolab/templates/ and os.path.join(__file__, '..', '..', '..', 'share', 'templates' paths are used to find templates.

• Use python UUID to convert binary objectGUID to string. This way dont need to chage cache db format. It doesn't look like value of uniqueid returned by from cache is used anywhere (needs to double check), so only forward conversion is needed.

Right, the complete fix might be very complicated because:

• we need different filters to list all roles, just non-managed roles or managed roles.

the alternative filters could be:
'(&(objectclass=ldapsubentry)(objectclass=nsComplexRoleDefinition))' - list all non-managed roles
'(&(objectclass=ldapsubentry)(objectclass=nsSimpleRoleDefinition))' - list all manged roles
'(&(objectclass=ldapsubentry)(objectclass=nsRoleDefinition))' - list all roles

• we need a separate code to implement user's roles listing (including editing managed roles). Currently, by default, Kolab User has a nsroledn attribute defined in WAP Settings. That implies that LDAP roles manaement follows logic of getting and changing regular multivalue LDAP attributes, but roles does not work like that. To set a role you need to add or remove nsroledn attribute, but checking role membership nsrole attribute should be used.

Getting rid of all smtplib calls and make all email sending via modules._sendmail

Fix typo

yet another attempt to create a better wrapper for smtplib.sendmail(). Now instantiation of smtplib object and connect are separated.

Log debug message after _sendmail.

Fix conner case exception handling.

Eliminating inconsistensies of sending mail from wallace. Now all the logic is in modules._sendmail() wrapper. Invitationpolicy module is using that wrapper and makes it responsible for re-submistion logic.

Make all pykolab log messages appear on level <8. This will isolate pykolab debug messages from 3rd partty libraries debug messages. -d debug -l 8 will print pykolab debug messages only, while -l debug -d 9 will print 3rd party libraries messages in addition to level 8 pykolab messages.

Make invitationpolicy module use smtplib sendmail wrapper _sendmail from modules, which now supports debug level 9 logging.

It turns out smtplib debug logging redirected to logger produces excessive new lines. This is attempt to mitigate the problem, make logs more pretty.

Fixing previous mistake. Need that for messages 9 and lower to be logged.

Need to catch imaplib debug output as well

Specifying a bit more detailed log message.

Fixing another wrong intendation

I think we need process id (PID) logged for any level log record, that makes it easy to associate and track wallace and other pykolab modules work flow.

Additionally, making wallace log.exception to log tracebacks

Fixing intendation

In addition LDAP trace_file needs to be specified. It is a file-like object define in logger.py Now we get ldap debug logs in pykolab.log file

This amendment tries to avoid cyruslib printing to non-available file descriptor, which causes traceback of wallace.

This is using the new file-like object from pykolab.logger. With this patch smtplib debug output gets redirected to logger rather than to stderr, which is not available in fork mode

Postfix verbose logging shows this:

Mar  8 20:03:17 prod-mail-proxy-1.pao1.server.com postfix/smtpd[22547]: connection established
Mar  8 20:03:17 prod-mail-proxy-1.pao1.server.com postfix/smtpd[22547]: master_notify: status 0
Mar  8 20:03:17 prod-mail-proxy-1.pao1.server.com postfix/smtpd[22547]: name_mask: resource
Mar  8 20:03:17 prod-mail-proxy-1.pao1.server.com postfix/smtpd[22547]: name_mask: software
Mar  8 20:03:17 prod-mail-proxy-1.pao1.server.com postfix/smtpd[22547]: connect from localhost[127.0.0.1]
Mar  8 20:03:17 prod-mail-proxy-1.pao1.server.com postfix/smtpd[22547]: match_list_match: localhost: no match
Mar  8 20:03:17 prod-mail-proxy-1.pao1.server.com postfix/smtpd[22547]: match_list_match: 127.0.0.1: no match
Mar  8 20:03:17 prod-mail-proxy-1.pao1.server.com postfix/smtpd[22547]: match_hostname: localhost ~? 127.0.0.0/8
Mar  8 20:03:17 prod-mail-proxy-1.pao1.server.com postfix/smtpd[22547]: match_hostaddr: 127.0.0.1 ~? 127.0.0.0/8
Mar  8 20:03:17 prod-mail-proxy-1.pao1.server.com postfix/smtpd[22547]: smtp_stream_setup: maxtime=300 enable_deadline=0
Mar  8 20:03:17 prod-mail-proxy-1.pao1.server.com postfix/smtpd[22547]: match_hostname: localhost ~? 127.0.0.0/8
Mar  8 20:03:17 prod-mail-proxy-1.pao1.server.com postfix/smtpd[22547]: match_hostaddr: 127.0.0.1 ~? 127.0.0.0/8
Mar  8 20:03:17 prod-mail-proxy-1.pao1.server.com postfix/smtpd[22547]: > localhost[127.0.0.1]: 220 mail.server.com ESMTP Postfix
Mar  8 20:03:17 prod-mail-proxy-1.pao1.server.com postfix/smtpd[22547]: watchdog_pat: 0x559232efc1c0
Mar  8 20:08:17 prod-mail-proxy-1.pao1.server.com postfix/smtpd[22547]: smtp_get: timeout
Mar  8 20:08:17 prod-mail-proxy-1.pao1.server.com postfix/smtpd[22547]: > localhost[127.0.0.1]: 421 4.4.2 mail.server.com Error: timeout exceeded
Mar  8 20:08:17 prod-mail-proxy-1.pao1.server.com postfix/smtpd[22547]: match_hostname: localhost ~? 127.0.0.0/8
Mar  8 20:08:17 prod-mail-proxy-1.pao1.server.com postfix/smtpd[22547]: match_hostaddr: 127.0.0.1 ~? 127.0.0.0/8
Mar  8 20:08:17 prod-mail-proxy-1.pao1.server.com postfix/smtpd[22547]: timeout after CONNECT from localhost[127.0.0.1]
Mar  8 20:08:17 prod-mail-proxy-1.pao1.server.com postfix/smtpd[22547]: disconnect from localhost[127.0.0.1]
Mar  8 20:08:17 prod-mail-proxy-1.pao1.server.com postfix/smtpd[22547]: master_notify: status 1
Mar  8 20:08:17 prod-mail-proxy-1.pao1.server.com postfix/smtpd[22547]: connection closed
Mar  8 20:08:17 prod-mail-proxy-1.pao1.server.com postfix/smtpd[22547]: watchdog_stop: 0x559232efc1c0

• To avoid double user/ prefix, check if user_mailbox_exists only by result_attribute

That is fine with me. I don't expect anything more fancy in this case.

I have changed the acls for anyone to lrs and that did the trick - don't see errors anymore. Wallace messages were dispatched.

No, user smith@domain.tld doesn't have any delegates in LDAP.

I see some odd behavior after applying this patch

Uncaught TypeError: me.has_attendees is not a function
    at update_event_confirm (https://mail.fsi.io/webmail/assets/plugins/calendar/calendar_ui.js:2512:29)
    at HTMLDivElement.eventDrop (https://mail.fsi.io/webmail/assets/plugins/calendar/calendar_ui.js:3799:9)
    at Calendar.trigger (https://mail.fsi.io/webmail/assets/plugins/calendar/lib/js/fullcalendar.js:1:10028)
    at trigger (https://mail.fsi.io/webmail/assets/plugins/calendar/lib/js/fullcalendar.js:3:21529)
    at eventDrop (https://mail.fsi.io/webmail/assets/plugins/calendar/lib/js/fullcalendar.js:3:23804)
    at HTMLDivElement.stop (https://mail.fsi.io/webmail/assets/plugins/calendar/lib/js/fullcalendar.js:3:4398)
    at t.(anonymous function).(anonymous function)._trigger (https://mail.fsi.io/webmail/assets/plugins/jqueryui/js/jquery-ui-1.10.4.custom.min.js:36:10036)
    at t.(anonymous function).(anonymous function)._trigger (https://mail.fsi.io/webmail/assets/plugins/jqueryui/js/jquery-ui-1.10.4.custom.min.js:36:29884)
    at t.(anonymous function).(anonymous function)._trigger (https://mail.fsi.io/webmail/assets/plugins/jqueryui/js/jquery-ui-1.10.4.custom.min.js:36:5029)
    at t.(anonymous function).(anonymous function)._mouseStop (https://mail.fsi.io/webmail/assets/plugins/jqueryui/js/jquery-ui-1.10.4.custom.min.js:36:23145)

and

Uncaught TypeError: me.has_attendees is not a function
    at event_edit_dialog (calendar_ui.js:697)
    at AgendaWeekView.select (calendar_ui.js:3743)
    at Calendar.trigger (fullcalendar.js:1)
    at trigger (fullcalendar.js:3)
    at reportSelection (fullcalendar.js:4)
    at HTMLDocument.<anonymous> (fullcalendar.js:2)
    at HTMLDocument.d (jquery.min.js:35)
    at HTMLDocument.dispatch (jquery.min.js:35)
    at HTMLDocument.r.handle (jquery.min.js:35)

is displayed several times in browser console.
Also I when I drag the event in attendees calendar I don't see any "saving..." box displayed in the corner as usual. I can't drag already dragged event once again.

Could be. I've tried to create new meeting and looking at the Caldav I see that ATTEDEE is:

ATTENDEE;CN=ks@fsi.io;PARTSTAT=NEEDS-ACTION;ROLE=REQ-PARTICIPANT;RSVP=TRUE:
 mailto:ks@fsi.io

So looks like SCHEDULE-AGENT is gone.

I was investigating a similar case - create event in Gmail and invite 2 attendees - one for user with Outlook and one for user in Roundcube. Both accept the invitation, but user who accepted it in Roundcube is not marked with a check box in Gmail.
it turns out, that Roundcube sends possibly wrong DTSTAMP in meeting acceptance iTip. Comparing responses I noticed that original invitation from Gmail has

METHOD:REQUEST
DTSTAMP:20170627T054332Z
CREATED:20170627T054331Z
LAST-MODIFIED:20170627T054331Z

Acceptance iTip coming from Outlook has:

METHOD:REPLY
CREATED:20170627T054452Z
DTSTAMP:20170627T054452Z
LAST-MODIFIED:20170627T054452Z

But Roundcube sends iTip with:

METHOD:REPLY
DTSTAMP:20170627T054331Z
CREATED:20170627T054331Z
LAST-MODIFIED:20170627T054331Z

It is one second behind from DTSTAMP value in invitation. If I send exactly the same response as Roundcube did, but modify DTSTAMP to be at least the same value as it was in invitation, then Gmail records the status of the attendee.
RFC5545 says that:

In the case of an iCalendar object that specifies a "METHOD"
property, this property differs from the "CREATED" and "LAST-
MODIFIED" properties.  These two properties are used to specify
when the particular calendar data in the calendar store was
created and last modified.  This is different than when the
iCalendar object representation of the calendar service
information was created or last modified.

I'm not sure I fully understand what they want to say here, but to me it looks like:

• DTSTAMP in REPLY should be later than it was in REQUEST
• DSTAMP should be different from CREATED and LAST-MODIFIED

no, I didn't