Page MenuHomePhorge
Create Task
Maniphest T7247

New Kolab install broken on Debian 11
Open, LowPublic
Actions

Description

Hello,

First of all thanks for the software! Have been using it for many years.

Not sure if this is the right/preferred place to report this, but it seems the Debian 11 packages have some broken dependencies (as a Debian 11 repo seems to exist I thought I'd try a fresh install of it). Following https://docs.kolab.org/installation-guide/debian-10.html , replacing 10.0 with 11.0:

Trying to install kolab:

root@kolab-deb11:~# apt-get update
Hit:1 http://deb.debian.org/debian bullseye InRelease
Get:2 http://deb.debian.org/debian bullseye-updates InRelease [44.1 kB]
Get:3 http://deb.debian.org/debian-security bullseye-security InRelease [48.4 kB]
Get:4 http://obs.kolabsys.com/repositories/Kolab:/16/Debian_11.0 ./ InRelease [1862 B]
Get:5 http://obs.kolabsys.com/repositories/Kolab:/16/Debian_11.0 ./ Sources [13.0 kB]
Get:6 http://obs.kolabsys.com/repositories/Kolab:/16/Debian_11.0 ./ Packages [21.7 kB]
Fetched 129 kB in 1s (152 kB/s)
Reading package lists... Done
root@kolab-deb11:~# apt install kolab
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 kolab-webclient : Depends: kolab-freebusy but it is not going to be installed
                   Depends: kolab-syncroton but it is not going to be installed
                   Depends: roundcubemail (>= 0.9) but it is not going to be installed
                   Depends: roundcubemail-plugin-contextmenu
                   Depends: roundcubemail-plugins-kolab (>= 3.0) but it is not installable
                   Depends: roundcubemail-skin-kolab
E: Unable to correct problems, you have held broken packages.

Explicitly adding roundcubemail seems to reduce the list:

root@kolab-deb11:~# apt install roundcubemail kolab
...
The following packages have unmet dependencies:
 kolab-webclient : Depends: kolab-freebusy but it is not going to be installed
                   Depends: roundcubemail-plugins-kolab (>= 3.0) but it is not installable
E: Unable to correct problems, you have held broken packages.

Adding kolab-freebusy as well leaves just the roundcubemail-plugins-kolab dep:

root@kolab-deb11:~# apt install kolab-freebusy roundcubemail kolab
...
The following packages have unmet dependencies:
 kolab-freebusy : Depends: roundcubemail-plugins-kolab but it is not installable
 kolab-webclient : Depends: roundcubemail-plugins-kolab (>= 3.0) but it is not installable
E: Unable to correct problems, you have held broken packages.

which can't be installed (as the error suggested):

root@kolab-deb11:~# apt install roundcubemail-plugins-kolab
...
Package roundcubemail-plugins-kolab is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'roundcubemail-plugins-kolab' has no installation candidate

Using aptitude -y install kolab, as from the documentation, gives similar issues:

root@kolab-deb11:~# aptitude -y install kolab
The following NEW packages will be installed:
  ... lots of packages ...
0 packages upgraded, 673 newly installed, 0 to remove and 0 not upgraded.
Need to get 340 MB of archives. After unpacking 1303 MB will be used.
The following packages have unmet dependencies:
 kolab : Depends: kolab-webclient but it is not installable
The following actions will resolve these dependencies:

     Keep the following packages at their current version:
1)     kolab [Not Installed]

The following NEW packages will be installed:
  binutils{a} binutils-common{a} binutils-x86-64-linux-gnu{a} cpp{a} cpp-10{a} dbus-user-session{a} dmidecode{a} dosfstools{a} fontconfig-config{a}
  fonts-dejavu-core{a} fonts-liberation{a} gawk{a} gcc{a} gcc-10{a} libasan6{a} libatomic1{a} libbinutils{a} libc-dev-bin{a} libc-devtools{a} libc6-dev{a}
  libcc1-0{a} libcrypt-dev{a} libctf-nobfd0{a} libctf0{a} libdeflate0{a} libfontconfig1{a} libfreetype6{a} libgcc-10-dev{a} libgd3{a} libglib2.0-0{a}
  libglib2.0-data{a} libgomp1{a} libicu67{a} libisl23{a} libitm1{a} libjbig0{a} libjpeg62-turbo{a} liblsan0{a} libmpc3{a} libmpfr6{a} libnsl-dev{a} libparted2{a}
  libpng16-16{a} libpolkit-agent-1-0{a} libpolkit-gobject-1-0{a} libquadmath0{a} libsasl2-modules-gssapi-mit{a} libsigsegv2{a} libterm-readline-gnu-perl{a}
  libtiff5{a} libtirpc-dev{a} libtsan0{a} libubsan1{a} libwebp6{a} libxml2{a} libxpm4{a} linux-libc-dev{a} make{a} manpages{a} manpages-dev{a} parted{a}
  policykit-1{a} shared-mime-info{a} ttf-bitstream-vera{a} ucf{a} xdg-user-dirs{a}
0 packages upgraded, 66 newly installed, 0 to remove and 0 not upgraded.
Need to get 69.3 MB of archives. After unpacking 252 MB will be used.

... lots of packages getting installed ...

Kolab isn't actually installed, running aptitude -y install kolab again gives most of the above output (with 607 new packages to be installed this time instead of 673 on first run, and ending with no more actual packages getting installed). setup-kolab isn't available either.

Expected behavior: package 'kolab' to actually install.

Details

Ticket Type
Task

Related Objects
Search...

Event Timeline

sevens created this task.Jan 27 2023, 5:00 PM
ghane mentioned this in T7255: New Kolab install broken on Ubuntu 22.04.Jan 30 2023, 12:06 PM
ghane lowered the priority of this task from Needs Triage to Low.Jan 30 2023, 1:17 PM
ghane subscribed.

Work for for newer Distributions are still in progress!

Debian 11 (Ubuntu 20.04+) for example has dropped support for Python2.7 there is more testing needed.
While still Debian 8 is under extended LTS support (https://www.debian.org/releases/).
Python2.7 compatibility is needed, also up to Debian 10, it's a python2.7 based Setup.

Help on testing and debugging errors is appreciated.

Use apt rdepends/depends to catch the needed packages from the other repositories.
Kolab-webclient work is still in progress, as there were changes in rouncdcube main design.
Did you try to install newer kolab-webclient with apt install kolab-webclient from
Kolab:/16:/Testing/Debian_11.0/
Kolab:/Winterfell/Debian_11.0/

I didn't try, so I didn't know, if all packages are available in the different repositories.

! WARNING ! Debian 11 / Ubuntu 20.04+ Installation is BETA/ALPHA and should only used for sneak preview, testing and debugging purposes. ! WARNING !

raptor2101 subscribed.Jan 30 2023, 5:11 PM
ghane closed this task as Resolved.EditedJan 30 2023, 6:04 PM
ghane claimed this task.

adding:
deb http://obs.kolabsys.com/repositories/Kolab:/16/Debian_11.0/ ./
deb http://obs.kolabsys.com/repositories/Kolab:/Winterfell/Debian_11.0/ ./
deb http://obs.kolabsys.com/repositories/Kolab:/16:/Testing/Debian_11.0/ ./
'changed the typo'

as apt sources
solves for me the depend errors

But caution it's no Production use. This is still Work in Progress

I haven't done the Installation, only checked dependencies with apt install kolab, try on your own risk.

ghane reopened this task as Open.Jan 30 2023, 6:05 PM
sevens added a comment.EditedJan 31 2023, 9:03 PM

Ok, here we go; this is a long one as I encountered quite a few issues, but hopefully much of it is useful...

TL;DR

Status: got the servers to run, can login to the kolab webadmin interface, didn't test the rest (yet)

To get it running at all:

  • Added repositories as noted in T7247#99868 (I tweaked the priority a bit as well, see below if that's actually needed (/etc/apt/preferences.d/kolab))
    • EDIT: except for the Debian 10.0, this was a mistake; see T7247#99901
  • Fixes and workarounds:
    • Fix Python version in shebang: sed -i 's;#! /usr/bin/python2;#! /usr/bin/python3;' /usr/sbin/setup-kolab
    • Install python-is-python3 package
    • Quick hack to fix wrong Python path:
# echo -e "\n\n# Temp work-around for Kolab packages\nimport sys\nsys.path.append('/usr/lib/python3.9/site-packages/')" >> /etc/python3.9/sitecustomize.py
# mv /usr/lib/python3/dist-packages/pykolab/* /usr/lib/python3.9/site-packages/pykolab/
# rmdir /usr/lib/python3/dist-packages/pykolab/
    • Call setup-kolab as: setup-kolab --with-php-ini=/etc/php/7.4/apache2/php.ini
  • Unknown/no solution found:
    • Can't get dirsrv-admin service (unknown if an actual issue, nor what a workaround/fix would be). Probably an 'issue' on Debian 10 as well

Intro

I didn't try installing from other reposes before (neither testing nor older versions), didn't think of it at the time :)

Below are relevant parts of my attempts, logs, debug output and fixes/workarounds (some proper, some ugly/temp ones).

I tested everything on Devuan '11' (chimaera) first (for the original report I checked Debian 11 as well, to ensure it wasn't Devuan-specific); I'm running this in a container on a host that doesn't have systemd, using containers that don't either seems to work better than those who do. AFAIK none of the issues below were due to a difference between both (where in doubt I checked on both). The few issues I ran into that were have been left out of this report. Let me know if any further testing on Debian 11 proper is needed as well.

For each individual issue I had to revert to a snapshot, apply the newly discovered fixes on top of the previous ones and run setup-kolab again from this 'fresh' system, as running it multiple times didn't work (safety checks failing due to part of the setup having been done already).

Summary

Issues discovered:

  • setup-kolab should call python3 in it's shebang, not python2
  • All Kolab-provided Debian packages should install in /usr/lib/python3/dist-packages/, not /usr/lib/python3.9/site-packages/. At least the kolab-conf Debian package already does this correctly.
  • No package providing the dirsrv-admin service available, not on Debian 10 either
  • wallaced has wrong shebang: calls python (which doesn't exist anymore).
    • Should either call python3 directly, or install python-is-python3 (according to Debian wiki the former is preferred).
  • php.ini should be looked for in /etc/php/7.4/apache2/php.ini

Additional notes and observations

  • kolab-conf installs the setup sub-package of pykolab, which seems a bit odd to me; I'd either install it all (or at least everything needed by setup-kolab, though not sure why not just letting it be handled by pykolab pkg anyway) or don't install anything of it at all.
  • I tested this all in a container. I have snapshots of just before installing everything; right after all packages where installed (and all workarounds/fixes have been applied), i.e. just before I did the setup-kolab call that finally worked; and a snapshot of right after everything got setup and starts. I can go back to them for further testing/digging up info if needed.
  • If setup-kolab fails after some steps have succeeded, it can be quite tricky to clean up after it so it can be ran again; I couldn't even get it to work after the first step (directory service) got set up. Using snapshots this wasn't much of a problem but I can't imagine how to get out of this when e.g. installing on a system that has other services setup already before installing Kolab (making a clean start/'system reinstall' quite a lot of work).
  • FYI: setup-kolab complains that it can't start guam on Devuan, as that code only supports systemd locations. But seeing the supported distros (all systemd using) that's probably intentional; just noting it for completeness :) I manually ensured it starts at boot.

Let me know if I can do some more testing, especially on the Python parts; I'm a long-time Python developer myself.

Log

Add additional repositories

Try using a mix of older repositories (Debian 10), testing (for deb 11) and 'Winterfell' (rolling release) (EDIT: for those replicating this, don't add the Debian 10.0 entries; see T7247#99901):

root@kolab-devu11:~# vim /etc/apt/sources.list.d/kolab.list
...
# Alternatives to get currently-missing deps.
deb [signed-by=/etc/apt/keyrings/kolab_community.pgp] http://obs.kolabsys.com/repositories/Kolab:/16/Debian_10.0/ ./
deb [signed-by=/etc/apt/keyrings/kolab_community.pgp] http://obs.kolabsys.com/repositories/Kolab:/Winterfell/Debian_11.0/ ./
deb [signed-by=/etc/apt/keyrings/kolab_community.pgp] http://obs.kolabsys.com/repositories/Kolab:/16:/Testing/Debian_11.0/ ./

and tweak priority a bit:

root@kolab-devu11:~# vim /etc/apt/preferences.d/kolab
Explanation: Prefer stable over other sources (used to get missing/wrong
Explanation: dependencies, due to Debian 11 support currently being in
Explanation: development/testing).
Explanation: Prefer Debian 10 over testing, if the version available there
Explanation: is good enough.  It's (probably) more stable.
Package: *
Pin: release n=Debian_10.0
Pin-Priority: 508

Explanation: Prefer testing over Winterfell; I assume it's probably closer to
Explanation: stable (in version at least).
Package: *
Pin: release l=Kolab:16:Testing
Pin-Priority: 505
Package: *
Pin: release l=Kolab:Winterfell
Pin-Priority: 502

Explanation: Prefer Kolab packages over Debian's
Package: *
Pin: origin obs.kolabsys.com
Pin-Priority: 510
root@kolab-devu11:~# apt install kolab
...
W: Sources disagree on hashes for supposely identical version '0.9.13-1' of 'guam:amd64'.
W: Sources disagree on hashes for supposely identical version '2.3.22-1~kolab1' of 'kolab-syncroton:amd64'.
Do you want to continue? [Y/n] y
...
NOTE: the "Sources disagree on hashes" warning is probably fine, we're mixing both distro versions and development tracks here, so it makes sense...

Answers to questions (defaulting to first, that's what the -y option from the Kolab install docs would pick as well):

  • Postfix Configuration: choose "No configuration"; assuming Kolab will config it later on...
  • Configuring cyrus-common: "Remove the mail and news spools?" (upon pkg uninstall it seems): No

All packages installed now!

Fix starting setup-kolab in the first place

Try to run setup-kolab:

root@kolab-devu11:~# setup-kolab
Traceback (most recent call last):
  File "/usr/sbin/setup-kolab", line 31, in <module>
    import pykolab
ImportError: No module named pykolab

which is weird, as pykolab is installed:

root@kolab-devu11:~# python3 -c 'import pykolab'
root@kolab-devu11:~#

checking /usr/sbin/setup-kolab shows the problem:

root@kolab-devu11:~# head -1 /usr/sbin/setup-kolab
#! /usr/bin/python2

Fix this and try again:

root@kolab-devu11:~# sed -i 's;#! /usr/bin/python2;#! /usr/bin/python3;' /usr/sbin/setup-kolab
root@kolab-devu11:~# setup-kolab
Traceback (most recent call last):
  File "/usr/sbin/setup-kolab", line 33, in <module>
    from pykolab.setup import Setup
  File "/usr/lib/python3/dist-packages/pykolab/setup/__init__.py", line 24, in <module>
    from pykolab.translate import _
ModuleNotFoundError: No module named 'pykolab.translate'

Investigate missing pykolab.translate:

  • pykolab seems to have installed only a little bit anyway:
root@kolab-devu11:~# ls /usr/lib/python3/dist-packages/pykolab/ -al
total 12
drwxr-xr-x  3 root root 4096 Jan 31 07:12 .
drwxr-xr-x 66 root root 4096 Jan 31 07:12 ..
drwxr-xr-x  3 root root 4096 Jan 31 07:13 setup
  • And no idea why this is entirely empty too:
root@kolab-devu11:~# apt-file list pykolab
root@kolab-devu11:~# # just to check that `apt-file list` isn't broken (e.g. due to Devuan):
root@kolab-devu11:~# apt-file list libc-bin
libc-bin: /etc/bindresvport.blacklist
...
  • Checking /var/lib/dpkg/info/pykolab.list *ONLY* shows Python files inside /usr/lib/python3.9/site-packages, nothing in (any) dist-packages/ dir...
root@kolab-devu11:~# grep -r /usr/lib/python3/dist-packages/pykolab/ /var/lib/dpkg/
/var/lib/dpkg/info/kolab-conf.list:/usr/lib/python3/dist-packages/pykolab/setup
... more lines, all pointing to `/var/lib/dpkg/info/kolab-conf.list` ...
  • Tested some more, e.g. removing from /usr/lib/python3/dist-packages/ entirely, didn't help
  • But the pykolab Debian package (which contains the rest of the Python pykolab package) gets installed in /usr/lib/python3.9/site-packages/pykolab/:
root@kolab-devu11:~# ls /usr/lib/python3.9/site-packages/pykolab/
__init__.py  auth  base.py  cli  conf  constants.py  errors.py  imap  imap_utf7.py  itip  logger.py  plugins  translate.py  translit.py  utils.py  wap_client  xml
  • which won't work. Quick test (after having temp moved /usr/lib/python3/dist-packages/pykolab/ to my home dir):
root@kolab-devu11:~# python3 -c 'import pykolab'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'pykolab
root@kolab-devu11:~# python3.9 -c 'import pykolab'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'pykolab'
  • Long story short: apparently, Debian packages should *only* install in dist-packages/, not in site-packages/ (as most/all other distros do). site-packages/ isn't even in sys.path; from Debian proper:
root@kolab-deb11:~# python3 -c 'import sys; print(sys.path)'
['', '/usr/lib/python39.zip', '/usr/lib/python3.9', '/usr/lib/python3.9/lib-dynload', '/usr/local/lib/python3.9/dist-packages', '/usr/lib/python3/dist-packages']
  • Ugly work-around (I don't care about user-installed Python's in /usr/(local/) for this container), after having restored /usr/lib/python3/dist-packages/pykolab/ again:
root@kolab-devu11:~# # Ensure all parts of the Python package are in the same dir
root@kolab-devu11:~# mv /usr/lib/python3/dist-packages/pykolab/* /usr/lib/python3.9/site-packages/pykolab/
root@kolab-devu11:~# rmdir /usr/lib/python3/dist-packages/pykolab/
root@kolab-devu11:~# echo -e "\n\n# Temp work-around for Kolab packages\nimport sys\nsys.path.append('/usr/lib/python3.9/site-packages/')" >> /etc/python3.9/sitecustomize.py
root@kolab-devu11:~# python3 -c 'import pykolab'
root@kolab-devu11:~# # Woohoo!
  • Test again:
root@kolab-devu11:~# setup-kolab -l debug
... got output now ...

Further dist-utils/ checks

Checking for other packages that install into site-packages:

root@kolab-devu11:~# grep -r --files-with-matches site-packages /var/lib/dpkg/
/var/lib/dpkg/info/python2.7-minimal.preinst
/var/lib/dpkg/info/python2.7.postinst
/var/lib/dpkg/info/kolab-xml.list
/var/lib/dpkg/info/python2.7.prerm
/var/lib/dpkg/info/kolab-cli.list
/var/lib/dpkg/info/kolab-xml.md5sums
/var/lib/dpkg/info/pykolab.list
/var/lib/dpkg/info/wallace.list
/var/lib/dpkg/info/pykolab.md5sums
/var/lib/dpkg/info/kolab-server.md5sums
/var/lib/dpkg/info/wallace.md5sums
/var/lib/dpkg/info/kolab-cli.md5sums
/var/lib/dpkg/info/kolab-saslauthd.list
/var/lib/dpkg/info/kolab-saslauthd.md5sums
/var/lib/dpkg/info/kolab-server.list

seems to yield only Kolab-provided packages (the python2.7* package matches are scripts that check for packages using the wrong location of site-packages/).

NOTE: just did a check (grep -r /usr/lib/python /var/lib/dpkg/ | grep --color=yes -- -packages) on a Debian 10 install of Kolab and everything in there installed into dist-packages/.

No dirsrv-admin

  • Going through the setup we're getting further now. Still getting a new error:
...
update-rc.d: error: initscript does not exist: /etc/init.d/dirsrv-admin
...

Broken wallaced

The Wallace part of the setup has some weird output:

...
Starting ClamAV daemon: clamd .
Restarting The Kolab Wallace server daemon is a content filtering daemon: wallacestart-stop-daemon: unable to start /usr/sbin/wallaced (No such file or directory)
 failed!
...

This seems to be from restarting the wallace service:

root@kolab:~# /usr/sbin/service wallace restart
Restarting The Kolab Wallace server daemon is a content filtering daemon: wallacestart-stop-daemon: unable to start /usr/sbin/wallaced (No such file or directory)
 failed!

which makes sense, as wallaced seems fully broken on Debian 11:

root@kolab:~# wallaced
bash: /usr/sbin/wallaced: /usr/bin/python: bad interpreter: No such file or directory

which is correct:

root@kolab:~# which python
root@kolab:~# which python2
/usr/bin/python2
root@kolab:~# which python3
/usr/bin/python3
NOTE: checked on actual Debian as well, just to be sure:
root@kolab-deb11:~# which python
root@kolab-deb11:~# which python2
root@kolab-deb11:~# which python3
/usr/bin/python3

https://wiki.debian.org/Python states: "NOTE: Debian 11 (bullseye) has removed the "python" package and the '/usr/bin/python' symlink due to the deprecation of Python 2. No packaged scripts should depend on the existence of '/usr/bin/python': if they do, that is a bug that should be reported to Debian. You can use the 'python-is-python3' or 'python-is-python2' packages to restore an appropriate '/usr/bin/python' symlink for third-party or legacy scripts."

So install:

root@kolab:~# apt install python-is-python3
...
root@kolab:~# which python
/usr/bin/python
root@kolab:~# python --version
Python 3.9.

as a quick workaround and now wallaced also starts correctly (though replacing the shebang is probably a better solution in hindsight).

PHP is too new

After that we get a PHP error:

...
2023-01-31 16:26:26,631 pykolab.setup ERROR [1634] Could not find PHP configuration file php.ini
...

Checking the code ( https://git.kolab.org/diffusion/P/browse/master/pykolab/setup/setup_php.py$97-110 ) none of the expected locations match; Debian 11 comes with PHP 7.4 so the proper path is /etc/php/7.4/apache2/php.ini

Luckily, this one is easy to work around though:

root@kolab:~# setup-kolab --with-php-ini=/etc/php/7.4/apache2/php.ini

That's it! Restoring the snapshot and running the above command resulted in a seemingly working setup; connecting and logging into the Kolab Webadmin (as cn=Directory Manager) works, didn't test anything else yet.

Devuan-specific fixes

Short summary of Devuan-specific fixes and workarounds, for those brave and/or crazy enough to try:

  • Patch /etc/init.d/dirsrv ( https://git.devuan.org/devuan/389-ds-base/issues/14)
  • Symlink: ln -s /etc/init.d/mariadb /etc/init.d/mysql
    • NOTE: this will result in various insserv: script mysql: service mariadb already provided! lines appearing during setup-kolab; seems harmless (and is gone after the temp workaround is removed again)
  • Install erlang-syntax-tools package, required by guam (didn't check why this wasn't installed; installing guam on Debian 11 *does* install this dep)
  • Use https://git.kolab.org/F9643183 as init script for guam; fix and enable it
  • Run setup-kolab --with-php-ini=/etc/php/7.4/apache2/php.ini
  • Remove temp /etc/init.d/mysql workaround
ghane removed ghane as the assignee of this task.Feb 3 2023, 1:02 AM
ghane added a subscriber: sicherha.

Thanks for the debugging

sorry Debian_10.0 should be Debian_11.0, this is a typo and could solve some of the issues.

remove deb http://obs.kolabsys.com/repositories/Kolab:/16/Debian_10.0/ ./
add  deb http://obs.kolabsys.com/repositories/Kolab:/16/Debian_11.0/ ./

dirsrv-admin is obsolete on Debian 11 and also Ubuntu 22.04, but still needed in Debian 10 for a proper 389-setup.
It is the old 389-admin package based on perl scripts. It's abandoned in newer 389ds versions, in favour of python3-lib389 and cockpit-389-ds package.

apt-key <...>

Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
Note: In your distribution this command is a no-op and can therefore be removed safely.`

Since Debian 11 and Ubuntu 20.04 they have no support for Python2 so using python3 is the the only choice.

python-is-python3 should be used, it will set the symlink python -> python3, older OS use the symlink python -> python2 (python3) ,too.

tilmankroesche subscribed.Feb 4 2023, 7:16 PM

Thx for the help, to setup Kolab on Debian 11. I've encountered another Error after upgrading to PHP 7.4:

PHP Fatal error: Uncaught Error: Call to undefined method SWIG\_p_Kolab__cDateTime::isValid() in /usr/share/roundcubemail/plugins/libkolab/lib/kolab_format.php:279

Unfortunately i haven't found a solution for this one yet.

tilmankroesche added a comment.Feb 5 2023, 1:19 PM

And yet another python exeption in wallace:

error: uncaptured python exception, closing channel <smtpd.SMTPChannel connected 127.0.0.1:37920 at 0x7f1fa0a390d0> (<class 'TypeError'>:process_message() got an unexpected keyword argument 'mail_options' [/usr/lib/python3.9/asyncore.py|read|83] [/usr/lib/python3.9/asyncore.py|handle_read_event|420] [/usr/lib/python3.9/asynchat.py|handle_read|171] [/usr/lib/python3.9/smtpd.py|found_terminator|386])

ghane added a comment.Feb 5 2023, 6:20 PM

Debian11 Install is testing....
check error with existing tasks on bug list.
As PHP and Python3 needs upgrades to newer versions, shipped with new distributions, there are still rough corners and known errors.

There was much activity on php parts since last package build.

Please checkout Git Version, as this is not a packaging error.

Your php error is from project : roundcube-plugins-kolab

https://git.kolab.org/diffusion/RPK/ -> roundcube-plugins-kolab
https://git.kolab.org/diffusion/

I could confirm the wrong pykolab install path, not "site-packages", should be /usr/lib/python3/dist-packages

Which wallace modules are configured in kolab.conf?
The error is from standard library, the error should be happened before. pykolab.log?

Wallace is based on pykolab, there are still some diffs waiting in git and are on discussion.

tilmankroesche added a comment.Feb 16 2023, 10:53 AM

Thx for the hints. I'l try the Git Version.

Wallace is configured with resources, invitationpolicy and footer.
I can't confirm, that the error has happend before.

with best regards

sevens added a comment.Feb 22 2023, 9:07 AM
In T7247#99901, @ghane wrote:

Thanks for the debugging

You're welcome, glad I could help!

sorry Debian_10.0 should be Debian_11.0, this is a typo and could solve some of the issues.

remove deb http://obs.kolabsys.com/repositories/Kolab:/16/Debian_10.0/ ./
add  deb http://obs.kolabsys.com/repositories/Kolab:/16/Debian_11.0/ ./

Ok, I've removed 10.0 (11.0 was already there, as the main repo) and did an apt update, apt upgrade cycle to upgrade packages from the wrong repo (and restarted the container).

I've also edited my log/report from above, referring to this comment in the places where the 10.0 repo was referenced, in case others are following it later on.

dirsrv-admin is obsolete on Debian 11 and also Ubuntu 22.04, but still needed in Debian 10 for a proper 389-setup.
It is the old 389-admin package based on perl scripts. It's abandoned in newer 389ds versions, in favour of python3-lib389 and cockpit-389-ds package.

Good to know, makes me feel more comfortable with ignoring the error :)

apt-key <...>

Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
Note: In your distribution this command is a no-op and can therefore be removed safely.`

Yeah, I got that one as well when following the instructions for Debian 10.0. I followed https://wiki.debian.org/DebianRepository/UseThirdParty#OpenPGP_certificate_distribution and added the [signed-by=...] part to my sources.list.d entries.

Since Debian 11 and Ubuntu 20.04 they have no support for Python2 so using python3 is the the only choice.

python-is-python3 should be used, it will set the symlink python -> python3, older OS use the symlink python -> python2 (python3) ,too.

Ok. Though, as a general note/information: from what I found when searching for this (from https://wiki.debian.org/Python) it seems Debian upstream highly recommends *not* using this package (and explicitly call python3 in your shebang/scripts instead of just python).

waadaa85 subscribed.Apr 8 2023, 9:08 PM
lennartack subscribed.Apr 23 2023, 2:00 PM
mroland subscribed.Apr 28 2023, 11:21 AM
waadaa85 added a comment.EditedMay 12 2023, 2:58 AM

Awesome job sevens !! Kolab Installation completed and kolab-setup ran successfully.
I've generated new cert-key pairs with letsencrypt.
Moving on :

  1. cyrus-imapd raised some minor errors : have to correct /etc/cyrus.conf socket paths for ptloader, lmtunix, notify to the right paths to /var/lib/cyrus/socket/...
  2. Wallaced is also running without error.
  3. kolab-webadmin is running too
  4. Unable to connect to roundcubemail, cursor spins without end, I'll dig this one after the guam problem (see next)
  5. Replaced the cert/key pairs in guam /etc/guam/sys.config file. And configured one server with two listeners.

Guam is running (not error in logs see below) but keep ballooning memory until the kernel kills it... (beam.smp filling all the available memory, kernel kills it, beam.smp restarts... and balloons again...

guam.service - Intelligent IMAP Reverse Proxy
     Loaded: loaded (/lib/systemd/system/guam.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2023-05-11 20:50:02 EDT; 1min 51s ago
   Main PID: 1859 (beam.smp)
      Tasks: 23 (limit: 4655)
     Memory: 3.0G
        CPU: 1min 55.976s
     CGroup: /system.slice/guam.service
             ├─1859 /usr/lib/erlang/lib/kolab_guam-0.9.13/bin/guam -Bd -- -root /usr/lib/erlang/lib/kolab_guam-0.9.13 -progname usr/lib/erlang/lib/kolab_guam-0.9.13/bin/guam -- -home /opt/k>
             ├─1925 erl_child_setup 1048576
             ├─1942 inet_gethost 4
             └─1943 inet_gethost 4

May 11 20:50:03 jules.conformit.ca guam[1859]: 20:50:03.370 [debug] Making listener "imap" with [{port,143},{imap_server,default},{rules,[{filter_groupware,[]}]},{tls_config,[{certfile,"/et>
May 11 20:50:03 jules.conformit.ca guam[1859]: 20:50:03.370 [debug] Making listener "imaps" with [{port,993},{implicit_tls,true},{imap_server,default},{rules,[{filter_groupware,[]}]},{tls_c>
May 11 20:50:03 jules.conformit.ca guam[1859]: 20:50:03.370 [debug] We have [{imap,{kolab_guam_listener,start_link,[imap,[{port,143},{imap_server,default},{rules,[{filter_groupware,[]}]},{t>
May 11 20:50:03 jules.conformit.ca guam[1859]: 20:50:03.371 [info] Starting listener "imap" on port 143 ([{keepalive,true},{reuseaddr,true},{active,false},inet6]) with 1 rules
May 11 20:50:03 jules.conformit.ca guam[1859]: 20:50:03.371 [debug] ImapConfig is [{host,"127.0.0.1"},{port,993},{tls,true}]
May 11 20:50:03 jules.conformit.ca guam[1859]: 20:50:03.371 [debug] Creating session pool of size 10 for listener <0.523.0>
May 11 20:50:03 jules.conformit.ca guam[1859]: 20:50:03.371 [info] Starting listener "imaps" on port 993 ([{keepalive,true},{reuseaddr,true},{active,false},inet6]) with 1 rules
May 11 20:50:03 jules.conformit.ca guam[1859]: 20:50:03.375 [debug] ImapConfig is [{host,"127.0.0.1"},{port,993},{tls,true}]
May 11 20:50:03 jules.conformit.ca guam[1859]: 20:50:03.375 [debug] Creating session pool of size 10 for listener <0.527.0>
May 11 20:50:03 jules.conformit.ca guam[1859]: 20:50:03.835 [debug] Lager installed handler lager_backend_throttle into lager_event

Beside the guam crashing memory, another peculiar element is the IPv6 clause at the end of the line. I don't know if its related.

[info] Starting listener "imaps" on port 993 ([{keepalive,true},{reuseaddr,true},{active,false},inet6]) with 1 rules

As IPv6 is disabled on my VM. How can I configure guam in sys.config to specify IPv4 ?

ghane added a comment.May 14 2023, 6:34 PM

@waadaa85 : Did you try to limit synchronous (+S) and asynchronous (+A) threads?

I guess where this could be set for beam.smp ( Erlang VM ) is:

/etc/guam/vm.args

Your roundcubemail trouble, could cause this.

Scenario:
Roundcube try to open and reopen a connection and causes lot of guam threads.
Without limitations this causes the crash.
Caching reopen old cached threads and before timeout is reached, the memory is filled with new threads.

ghane added a comment.May 16 2023, 7:27 PM

wallace error mentioned here: T7247#100099

error: uncaptured python exception, closing channel <smtpd.SMTPChannel connected 127.0.0.1:37920 at 0x7f1fa0a390d0> (<class 'TypeError'>:process_message() got an unexpected >keyword argument 'mail_options' [/usr/lib/python3.9/asyncore.py|read|83] [/usr/lib/python3.9/asyncore.py|handle_read_event|420] [/usr/lib/python3.9/asynchat.py|handle_read|171] >[/usr/lib/python3.9/smtpd.py|found_terminator|386])

I run into same error !

Python module: smtpd, asyncore deleted in Python 3.12
but recommended replacement aiosmtpd and asyncio not Python 2.7 compatible.

process_message is smtpd module

( https://bugs.python.org/issue35837 )

wallace/__init__.py line 477:

def process_message(self, peer, mailfrom, rcpttos, data, **kwargs)

results in new error:

Traceback (most recent call last):

File "/opt/pykolab/wallace/modules.py", line 158, in _sendmail
  _response = smtp.sendmail(sender, recipients, msg)
File "/usr/lib/python3.9/smtplib.py", line 865, in sendmail
  (code, resp) = self.mail(from_addr, esmtp_opts)
File "/usr/lib/python3.9/smtplib.py", line 538, in mail
  self.putcmd("mail", "FROM:%s%s" % (quoteaddr(sender), optionlist))
File "/usr/lib/python3.9/smtplib.py", line 153, in quoteaddr
  if addrstring.strip().startswith('<'):
 
AttributeError: 'list' object has no attribute 'strip'
sevens added a comment.EditedJun 1 2023, 11:17 AM

So I finally had some time to look into this some more. First, thanks to all others for their notes and further progress, this helped me today as well. I only had some of the issues others had, possibly as they have been fixed in the packages/upstream in the meantime, and hit some issues not yet mentioned here as well. What follows is an extensive log of and some notes on the relevant parts of my updates, changes and fixes since my last comment. Hoping this helps others further along as well.

TL;DR:

  • Got both roundcube and chwala working and guam seems to run fine too.
  • Both cyrus-imapd and postfix require an SSL cert to be created and configured.
  • php-curl needs to be installed.
  • Haven't noticed any wallace issues and IPv6 localhost seems to get used as well.
  • Sending mail still broken, didn't investigate.

Upgraded all packages (it's been a while)

root@kolab:~# apt update && apt upgrade

Had config file conflicts:

  • /etc/kolab/kolab.conf: needed manual merge (to keep the passwords, domain names, etc). Merged by opening a shell (Z option) and running vimdiff $DPKG_CONFFILE_OLD $DPKG_CONFFILE_NEW
  • /etc/roundcubemail/libkolab.inc.php: installed package maintainer's version

Create a test-user so we can test some more stuff

Following https://docs.kolab.org/howtos/kolab-webadmin-create-user.html#howto-kolab-webadmin-create-user , created a testuser for myself, added to kolab-admin role.

Check notes and issues from T7247#102095 (waadaa85)

cryus-imapd:

  • Quick and dirty (/var/lib/imap/ dir doesn't even exist):
root@kolab:~# sed -i 's#/var/lib/imap/#/var/lib/cyrus/#' /etc/cyrus.conf
root@kolab:~# service cyrus-imapd restart

Wallace: AFAICT I have no issues either. I checked:

  • ps aux | grep -i wall, had 6 wallaced processes
  • GLOBIGNORE='/var/log/apt'; grep -r wallace /var/log/* | less, showed no errors. It did have a warning though:
/var/log/kolab/pykolab.log:2023-06-01 02:34:50,943 pykolab.wallace WARNING [20595] Exiting Wallace_Heartbeat, 0

Roundcube: trying to login fails with error: "Connection to storage server failed." The error appears fairly quickly (5-10 seconds?) and no spinning cursor/other loading indications in the meantime.

Guam certs: didn't check at this point, added self-signed cert later on (see below)

Attempt to get roundcube working

Check logs, processes:

root@kolab:/var/log/roundcubemail# tail /var/log/roundcubemail/errors.log
[01-Jun-2023 03:14:50,696620 +0200]: <ndlii23c> IMAP Error: Login failed for peter.sevens@example.com against localhost from 192.168.1.8. Empty startup greeting (localhost:143) in /usr/share/roundcubemail/program/lib/Roundcube/rcube_imap.php on line 211 (POST /roundcubemail/?_task=login&_action=login)
... same line for 3 later timestamps ...
root@kolab:/var/log/roundcubemail# ps aux | grep guam
... actually running (PIDs 1569 and 1581) ...

Found possibly related upstream bugreport ( https://github.com/roundcube/roundcubemail/issues/6016 ), trying some options:

  • Change $config['default_host'] from tls://localhost to ssl://localhost: no effect
  • Change $config['default_port'] from 143 to 993: no effect (both with and without the change from above)

Test basic SSL connection:

root@kolab:~# openssl s_client -connect localhost:993 -tls1_2
CONNECTED(00000003)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 188 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1685591278
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
---

Possibly not working correctly due to non-existing cert. Test with self-signed certificate:

root@kolab:/etc/roundcubemail# openssl req -x509 -new -keyout /etc/ssl/private/cyrus-imapd.key -out /etc/ssl/private/cyrus-imapd.pem -days 365 -nodes
Country Name (2 letter code) [AU]:NL
State or Province Name (full name) [Some-State]:F
Locality Name (eg, city) []:G
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Kolab Example
Organizational Unit Name (eg, section) []:x
Common Name (e.g. server FQDN or YOUR name) []:kolab.example.com
Email Address []:peter.sevens@kolab.example.com
root@kolab:/etc/roundcubemail# chmod 640 /etc/ssl/private/cyrus-imapd.key
root@kolab:/etc/roundcubemail# chown root:ssl-cert /etc/ssl/private/cyrus-imapd.key
root@kolab:/etc/roundcubemail# usermod -a -G ssl-cert cyrus
root@kolab:/etc/roundcubemail# groups cyrus
cyrus : mail sasl ssl-cert
root@kolab:/etc/roundcubemail# service cyrus-imapd restart
Restarting Cyrus IMAPd: cyrmaster failed!
root@kolab:/etc/roundcubemail# service cyrus-imapd stop
root@kolab:/etc/roundcubemail# service cyrus-imapd start
root@kolab:~# openssl s_client -connect localhost:993 -tls1_2
... still same error ...

Configure Guam to use generated key file as well (as suggested by T7247#102095 , config options from https://docs.kolab.org/about/guam/configuration.html ):

root@kolab:/etc/guam# vim sys.config
...
                    listeners, [
                        ...
                        imap, [
                            tls_config, [
                                ...
                                    { certfile, ... },	# Added trailing comma
                                    { keyfile, "/etc/ssl/private/cyrus-imapd.key" }
                                ...
                            ]
                        ...
                        imaps, [
                            tls_config, [
                                ...
                                    { certfile, ... },	# Added trailing comma
                                    { keyfile, "/etc/ssl/private/cyrus-imapd.key" }
                                ...
                            ]
                        ...

root@kolab:/etc/guam# service cyrus-imapd restart
root@kolab:/etc/guam# service guam restart
root@kolab:/etc/guam# openssl s_client -connect localhost:993 -tls1_2
... same ...

Get some saner SMTP/IMAP testing options

Before diving deeper get some more specific, saner testing options:

root@kolab:/etc/roundcubemail# apt install cyrus-clients
root@kolab:/etc/roundcubemail# /usr/lib/cyrus/bin/smtptest localhost -t ''
S: 220 kolab.localdomain ESMTP Postfix (Debian/GNU)
C: EHLO smtptest
S: 250-kolab.localdomain
S: 250-PIPELINING
S: 250-SIZE 10240000
S: 250-VRFY
S: 250-ETRN
S: 250-STARTTLS
S: 250-ENHANCEDSTATUSCODES
S: 250-8BITMIME
S: 250-DSN
S: 250-SMTPUTF8
S: 250 CHUNKING
C: STARTTLS
S: 454 4.7.0 TLS not available due to local problem
Authentication failed. no mechanism available
Security strength factor: 0
^CC: QUIT
Connection closed.

Try to get Cyrus to work (in general and SSL/TLS specific)

root@kolab:/var/lib/cyrus/log# kolab -d 5 lm
2023-06-01 04:49:39,462 pykolab.conf DEBUG [4215] Setting options from configuration file
2023-06-01 04:49:39,556 pykolab.imap ERROR [4215] Login to Cyrus IMAP server failed: CYRUSError(10, 'LOGIN', "b'Invalid user'")
Traceback (most recent call last):
  File "/usr/sbin/kolab", line 41, in <module>
    kolab = Cli()
  File "/usr/lib/python3.9/site-packages/pykolab/cli/__init__.py", line 77, in __init__
    commands.execute('_'.join(to_execute))
  File "/usr/lib/python3.9/site-packages/pykolab/cli/commands.py", line 135, in execute
    commands[cmd_name]['function'](conf.cli_args, kw)
  File "/usr/lib/python3.9/site-packages/pykolab/cli/cmd_list_mailboxes.py", line 97, in execute
    folders.extend(imap.lm(imap_utf7.encode(search)))
  File "/usr/lib/python3.9/site-packages/pykolab/imap/__init__.py", line 1271, in lm
    return self.imap.lm(*args, **kw)
  File "/usr/lib/python3.9/site-packages/cyruslib.py", line 687, in lm
    self.__prepare('LIST')
  File "/usr/lib/python3.9/site-packages/cyruslib.py", line 547, in __prepare
    self.__doexception(command, self.ERROR.get("NOAUTH")[1])
  File "/usr/lib/python3.9/site-packages/cyruslib.py", line 533, in __doexception
    self.__doraise( function.upper(), msg )
  File "/usr/lib/python3.9/site-packages/cyruslib.py", line 542, in __doraise
    raise CYRUSError( idError[0], mode, msg )
cyruslib.CYRUSError: (99, 'LIST', 'Connection is not authenticated')
root@kolab:/var/lib/cyrus/log# testsaslauthd -u cyrus-admin -p PASSWORD
0: OK "Success."
root@kolab:/var/log# /usr/lib/cyrus/bin/imtest localhost -t ''
S: * OK [CAPABILITY IMAP4rev1 STARTTLS LITERAL+ ID ENABLE SASL-IR LOGINDISABLED] kolab Cyrus IMAP 3.2.6-Debian-3.2.6-2+deb11u2 server ready
C: S01 STARTTLS
S: S01 OK Begin TLS negotiation now
SSL_connect error -1
SSL session removed
failure: TLS negotiation failed!

Use proper cert file (instead of PEM) in case it helps:

root@kolab:/var/lib/cyrus/log# openssl x509 -outform der -in /etc/ssl/private/cyrus-imapd.pem -out /etc/ssl/private/cyrus-imapd.crt
root@kolab:/var/lib/cyrus/log# vim /etc/guam/sys.config
root@kolab:/var/lib/cyrus/log# vim /etc/imapd.conf	# Changed cert key to .key file (was .pem), not sure if needed
root@kolab:/var/lib/cyrus/log# service guam stop
root@kolab:/var/lib/cyrus/log# service guam start
root@kolab:/var/lib/cyrus/log# service cyrus-imapd stop
root@kolab:/var/lib/cyrus/log# service cyrus-imapd start

No progress...

Get proper cyrus-imapd logging:

root@kolab:/var/lib/cyrus/log# apt install rsyslog
root@kolab:/var/lib/cyrus/log# service rsyslog start
root@kolab:/var/lib/cyrus/log# service cyrus-imapd stop
root@kolab:/var/lib/cyrus/log# service cyrus-imapd start

We now have /var/log/mail.* files!

Trying to get basic IMAP login to work

root@kolab:/var/log/roundcubemail# /usr/lib/cyrus/bin/imtest localhost -t '' -u peter.sevens@kolab.example.com -a peter.sevens@kolab.example.com -w test123
S: * OK [CAPABILITY IMAP4rev1 STARTTLS LITERAL+ ID ENABLE SASL-IR LOGINDISABLED] kolab Cyrus IMAP 3.2.6-Debian-3.2.6-2+deb11u2 server ready
C: S01 STARTTLS
S: S01 OK Begin TLS negotiation now
verify error:num=18:self signed certificate
TLS connection established: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
C: C01 CAPABILITY
S: * CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxten QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SEARCH=FUZZY SORT SORT=MODSEQ SORT=DISPLAY SORT=UID THREAD=ORDEREDSUBJECT THREAD=REFERENCES THREAD=REFS ANNOTATE-EXPERIMENT-1 METADATA LIST-EXTENDED LIST-STATUS LIST-MYRIGHTS LIST-METADATA WITHIN QRESYNC SCAN XLIST XMOVE MOVE SPECIAL-USE CREATE-SPECIAL-USE DIGEST=SHA1 X-REPLICATION STATUS=SIZE OBJECTID SAVEDATE X-CREATEDMODSEQ PREVIEW=FUZZY URLAUTH URLAUTH=BINARY AUTH=PLAIN AUTH=LOGIN SASL-IR COMPRESS=DEFLATE X-QUOTA=STORAGE X-QUOTA=MESSAGE X-QUOTA=X-ANNOTATION-STORAGE X-QUOTA=X-NUM-FOLDERS IDLE
S: C01 OK Completed
C: A01 AUTHENTICATE PLAIN cGV0ZXIuc2V2ZW5zQGtvbGFiLnRlc3QuZWhyYS5sYW4AcGV0ZXIuc2V2ZW5zQGtvbGFiLnRlc3QuZWhyYS5sYW4AdGVzdDEyMw==
S: A01 NO authentication failure
Authentication failed. generic failure
Security strength factor: 256
root@kolab:/var/log# tail /var/log/mail.err
Jun  1 05:40:14 kolab cyrus/imaps[8193]: SASL bad userid authenticated
Jun  1 05:40:17 kolab cyrus/imaps[8194]: ptload(): can't connect to ptloader server: No such file or directory
Jun  1 05:40:17 kolab cyrus/imaps[8194]: ptload completely failed: unable to canonify identifier: cyrus-admin
Jun  1 05:40:17 kolab cyrus/imaps[8194]: SASL bad userid authenticated
Jun  1 05:43:30 kolab cyrus/imaps[8253]: ptload(): can't connect to ptloader server: No such file or directory
Jun  1 05:43:30 kolab cyrus/imaps[8253]: ptload completely failed: unable to canonify identifier: peter.sevens@kolab.example.com
Jun  1 05:43:30 kolab cyrus/imaps[8253]: SASL bad userid authenticated
Jun  1 05:44:09 kolab cyrus/imaps[8286]: ptload(): can't connect to ptloader server: No such file or directory
Jun  1 05:44:09 kolab cyrus/imaps[8286]: ptload completely failed: unable to canonify identifier: peter.sevens@kolab.example.com
Jun  1 05:44:09 kolab cyrus/imaps[8286]: SASL bad userid authenticated

Already changed that in /etc/cyrus.conf, check further:

root@kolab:/etc/guam# ls -al /var/lib/cyrus/socket/ptsock	# File exists?
srwxrwxrwx 1 root root 0 Jun  1 05:29 /var/lib/cyrus/socket/ptsock
root@kolab:/etc/guam# grep -r ptload /etc
/etc/imapd.conf:ptloader_sock: /var/lib/imap/socket/ptsock
/etc/cyrus.conf:    ptloader    cmd="ptloader"  listen="/var/lib/cyrus/socket/ptsock"   prefork=0
root@kolab:/etc/guam# sed -i 's#/var/lib/imap/#/var/lib/cyrus/#' /etc/imapd.conf
root@kolab:/etc/guam# service cyrus-imapd stop && service cyrus-imapd start

and test again:

root@kolab:/etc/guam# /usr/lib/cyrus/bin/imtest localhost -t '' -u peter.sevens@kolab.example.com -a peter.sevens@kolab.example.com -w test123
S: * OK [CAPABILITY IMAP4rev1 STARTTLS LITERAL+ ID ENABLE SASL-IR LOGINDISABLED] kolab Cyrus IMAP 3.2.6-Debian-3.2.6-2+deb11u2 server ready
C: S01 STARTTLS
S: S01 OK Begin TLS negotiation now
verify error:num=18:self signed certificate
TLS connection established: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
C: C01 CAPABILITY
S: * CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxten QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SEARCH=FUZZY SORT SORT=MODSEQ SORT=DISPLAY SORT=UID THREAD=ORDEREDSUBJECT THREAD=REFERENCES THREAD=REFS ANNOTATE-EXPERIMENT-1 METADATA LIST-EXTENDED LIST-STATUS LIST-MYRIGHTS LIST-METADATA WITHIN QRESYNC SCAN XLIST XMOVE MOVE SPECIAL-USE CREATE-SPECIAL-USE DIGEST=SHA1 X-REPLICATION STATUS=SIZE OBJECTID SAVEDATE X-CREATEDMODSEQ PREVIEW=FUZZY URLAUTH URLAUTH=BINARY AUTH=PLAIN AUTH=LOGIN SASL-IR COMPRESS=DEFLATE X-QUOTA=STORAGE X-QUOTA=MESSAGE X-QUOTA=X-ANNOTATION-STORAGE X-QUOTA=X-NUM-FOLDERS IDLE
S: C01 OK Completed
C: A01 AUTHENTICATE PLAIN cGV0ZXIuc2V2ZW5zQGtvbGFiLnRlc3QuZWhyYS5sYW4AcGV0ZXIuc2V2ZW5zQGtvbGFiLnRlc3QuZWhyYS5sYW4AdGVzdDEyMw==
S: A01 NO authentication failure
Authentication failed. generic failure
Security strength factor: 256
^CC: Q01 LOGOUT
Connection closed.
root@kolab:/etc/guam# tail /var/log/mail.err
Jun  1 05:44:09 kolab cyrus/imaps[8286]: ptload(): can't connect to ptloader server: No such file or directory
Jun  1 05:44:09 kolab cyrus/imaps[8286]: ptload completely failed: unable to canonify identifier: peter.sevens@kolab.example.com
Jun  1 05:44:09 kolab cyrus/imaps[8286]: SASL bad userid authenticated
Jun  1 05:45:19 kolab cyrus/imaps[8291]: ptload(): can't connect to ptloader server: No such file or directory
Jun  1 05:45:19 kolab cyrus/imaps[8291]: ptload completely failed: unable to canonify identifier: cyrus-admin
Jun  1 05:45:19 kolab cyrus/imaps[8291]: SASL bad userid authenticated
Jun  1 05:49:19 kolab cyrus/ptloader[8385]: No entries found
Jun  1 05:49:19 kolab cyrus/imaps[8377]: ptload(): bad response from ptloader server: identifier not found
Jun  1 05:49:19 kolab cyrus/imaps[8377]: ptload completely failed: unable to canonify identifier: peter.sevens@kolab.example.com
Jun  1 05:49:19 kolab cyrus/imaps[8377]: SASL bad userid authenticated

Try different user:

root@kolab:/etc/guam# /usr/lib/cyrus/bin/imtest localhost -t '' -u sevens -a sevens -w test123
S: * OK [CAPABILITY IMAP4rev1 STARTTLS LITERAL+ ID ENABLE SASL-IR LOGINDISABLED] kolab Cyrus IMAP 3.2.6-Debian-3.2.6-2+deb11u2 server ready
C: S01 STARTTLS
S: S01 OK Begin TLS negotiation now
verify error:num=18:self signed certificate
TLS connection established: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
C: C01 CAPABILITY
S: * CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxten QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SEARCH=FUZZY SORT SORT=MODSEQ SORT=DISPLAY SORT=UID THREAD=ORDEREDSUBJECT THREAD=REFERENCES THREAD=REFS ANNOTATE-EXPERIMENT-1 METADATA LIST-EXTENDED LIST-STATUS LIST-MYRIGHTS LIST-METADATA WITHIN QRESYNC SCAN XLIST XMOVE MOVE SPECIAL-USE CREATE-SPECIAL-USE DIGEST=SHA1 X-REPLICATION STATUS=SIZE OBJECTID SAVEDATE X-CREATEDMODSEQ PREVIEW=FUZZY URLAUTH URLAUTH=BINARY AUTH=PLAIN AUTH=LOGIN SASL-IR COMPRESS=DEFLATE X-QUOTA=STORAGE X-QUOTA=MESSAGE X-QUOTA=X-ANNOTATION-STORAGE X-QUOTA=X-NUM-FOLDERS IDLE
S: C01 OK Completed
C: A01 AUTHENTICATE PLAIN c2V2ZW5zAHNldmVucwB0ZXN0MTIz
S: A01 OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxten QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SEARCH=FUZZY SORT SORT=MODSEQ SORT=DISPLAY SORT=UID THREAD=ORDEREDSUBJECT THREAD=REFERENCES THREAD=REFS ANNOTATE-EXPERIMENT-1 METADATA LIST-EXTENDED LIST-STATUS LIST-MYRIGHTS LIST-METADATA WITHIN QRESYNC SCAN XLIST XMOVE MOVE SPECIAL-USE CREATE-SPECIAL-USE DIGEST=SHA1 X-REPLICATION STATUS=SIZE OBJECTID SAVEDATE X-CREATEDMODSEQ PREVIEW=FUZZY URLAUTH URLAUTH=BINARY LOGINDISABLED COMPRESS=DEFLATE X-QUOTA=STORAGE X-QUOTA=MESSAGE X-QUOTA=X-ANNOTATION-STORAGE X-QUOTA=X-NUM-FOLDERS IDLE] Success (tls protection) SESSIONID=<kolab-1685598640-8377-1-6369895154409993212>
Authenticated.
Security strength factor: 256

Woohoo! Roundcube login now works as well!!

Try to get Postfix/SMTP to work (in general and with SSL/TLS)

Got similar TLS errors, e.g.:

root@kolab:/var/log# cat syslog
...
Jun  1 05:20:15 kolab postfix/smtpd[6984]: warning: No server certs available. TLS won't be enabled
...

Basic test:

root@kolab:/etc/guam# /usr/lib/cyrus/bin/smtptest localhost -t '' -u sevens -a sevens -w test123
S: 220 kolab.localdomain ESMTP Postfix (Debian/GNU)
C: EHLO smtptest
S: 250-kolab.localdomain
S: 250-PIPELINING
S: 250-SIZE 10240000
S: 250-VRFY
S: 250-ETRN
S: 250-STARTTLS
S: 250-ENHANCEDSTATUSCODES
S: 250-8BITMIME
S: 250-DSN
S: 250-SMTPUTF8
S: 250 CHUNKING
C: STARTTLS
S: 454 4.7.0 TLS not available due to local problem
Authentication failed. no mechanism available
Security strength factor: 0
^CC: QUIT
Connection closed.
root@kolab:/etc/guam#

Fix TLS (smtpd_tls_auth_only conf option is already set to yes but no TLS is configured...):

root@kolab:/etc/postfix# postfix tls deploy-server-cert /etc/ssl/private/cyrus-imapd.pem /etc/ssl/private/cyrus-imapd.key
postfix/postfix-tls-script: New TLS private key and certificate deployed.
postfix/postfix-tls-script: Run the command:
postfix/postfix-tls-script:   # postfix reload
postfix/postfix-tls-script: if you want the new settings to take effect immediately.
root@kolab:/etc/postfix# postfix reload
postfix/postfix-script: refreshing the Postfix mail system
root@kolab:/etc/postfix# /usr/lib/cyrus/bin/smtptest localhost -t '' -u sevens -a sevens -w test123
S: 220 kolab.localdomain ESMTP Postfix (Debian/GNU)
C: EHLO smtptest
S: 250-kolab.localdomain
S: 250-PIPELINING
S: 250-SIZE 10240000
S: 250-VRFY
S: 250-ETRN
S: 250-STARTTLS
S: 250-ENHANCEDSTATUSCODES
S: 250-8BITMIME
S: 250-DSN
S: 250-SMTPUTF8
S: 250 CHUNKING
C: STARTTLS
S: 220 2.0.0 Ready to start TLS
verify error:num=18:self signed certificate
TLS connection established: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
C: EHLO smtptest
S: 250-kolab.localdomain
S: 250-PIPELINING
S: 250-SIZE 10240000
S: 250-VRFY
S: 250-ETRN
S: 250-AUTH PLAIN LOGIN
S: 250-ENHANCEDSTATUSCODES
S: 250-8BITMIME
S: 250-DSN
S: 250-SMTPUTF8
S: 250 CHUNKING
C: AUTH PLAIN c2V2ZW5zAHNldmVucwB0ZXN0MTIz
S: 235 2.7.0 Authentication successful
Authenticated.
Security strength factor: 256
^CC: QUIT
Connection closed.

Test sending mail

Tested:

  • Sent a test mail to myself, through roundcube, to different alias address (Kolab address) than the one I'm logged in with
  • Sent a test mail to my main Kolab address too
  • Sent a mail to my personal address (on personal domain)

Results:

  • Sending mails showed no errors (it did before the Postfix TLS fix)
  • The test mails sent to the kolab addresses don't appear in my Inbox though, but they do appear in Sent.
  • The test mail sent to my personal address didn't arrive either, but I didn't expect it to

Didn't investigate this issue further as sending mails is the least important part for me to get working atm.

Chwala

Quick test: can't login. Also got an error with the 'Files' tab in roundcube

Related log:

root@kolab:/var/log/chwala# tail errors.log
[01-Jun-2023 06:25:01,689052 +0200]: <9ceh4bv9> IMAP Error: Login failed for peter.sevens@example.com against localhost from 127.0.0.1. AUTHENTICATE PLAIN: authentication failure in /usr/share/roundcubemail/program/lib/Roundcube/rcube_imap.php on line 211 (POST /chwala/api/?version=2&method=authenticate)
[01-Jun-2023 06:25:31,124442 +0200]: <39s0vd71> IMAP Error: Login failed for peter.sevens@example.com against localhost from 127.0.0.1. AUTHENTICATE PLAIN: authentication failure in /usr/share/roundcubemail/program/lib/Roundcube/rcube_imap.php on line 211 (POST /chwala/api/?version=2&method=authenticate)

Fix PHP cURL extension error

When testing the 'Files' tab in roundcube I noticed a PHP error:

root@kolab:/var/log/roundcubemail# tail errors.log
[01-Jun-2023 06:29:23,286985 +0200]: <i1vuf13h> PHP Error: cURL extension not available in /usr/share/roundcubemail/vendor/pear/http_request2/HTTP/Request2/Adapter/Curl.php on line 179 (GET /roundcubemail/o1BNrnZRUUzSsXz4/?_task=files)
... same line repeated multiple times ...
[01-Jun-2023 06:29:23,289970 +0200]: <i1vuf13h> PHP Error: cURL extension not available in /usr/share/roundcubemail/plugins/kolab_files/lib/kolab_files_engine.php on line 1729 (GET /roundcubemail/o1BNrnZRUUzSsXz4/?_task=files)

Fix:

root@kolab:/var/log/roundcubemail# apt install php-curl
...
root@kolab:/var/log/roundcubemail# service apache2 restart

This fixed both the 'Files' tab in roundcube as the Chwala direct login.

Guam

Errors I had earlier today (see bottom) seem to be gone now as well, newest one is ~2h old at moment of writing. Probably fixed by one of the fixes above (e.g. cyrus now working).

Remaining issues others had

With regards to IPv4/IPv6:

  • My container internally has IPv6 for localhost, no IPv6 to the host nor on the physical network.
  • imaps, imap2, ldap and ldaps seem to only listen on the IPv6 localhost IP, smtp and sieve on both 4 and 6. This seems to work fine for Kolab's internal communication of the things I tested.
  • Haven't tested any IPv4 connectivity from outside the container except to the webserver (webadmin, roundcube, chwala, etc). Connection was from my PC (container runs on my local server, which is a physically different machine)
  • For completeness/reference:
root@kolab:/var/log/guam/log# netstat -t -l
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:39327           0.0.0.0:*               LISTEN
tcp        0      0 localhost:mysql         0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:epmd            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:smtp            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:sieve           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:submission      0.0.0.0:*               LISTEN
tcp        0      0 localhost:10026         0.0.0.0:*               LISTEN
tcp        0      0 localhost:10027         0.0.0.0:*               LISTEN
tcp        0      0 localhost:10024         0.0.0.0:*               LISTEN
tcp        0      0 localhost:10025         0.0.0.0:*               LISTEN
tcp        0      0 localhost:9993          0.0.0.0:*               LISTEN
tcp6       0      0 localhost:10024         [::]:*                  LISTEN
tcp6       0      0 [::]:epmd               [::]:*                  LISTEN
tcp6       0      0 [::]:ldap               [::]:*                  LISTEN
tcp6       0      0 [::]:smtp               [::]:*                  LISTEN
tcp6       0      0 [::]:http               [::]:*                  LISTEN
tcp6       0      0 [::]:sieve              [::]:*                  LISTEN
tcp6       0      0 [::]:imap2              [::]:*                  LISTEN
tcp6       0      0 [::]:imaps              [::]:*                  LISTEN
tcp6       0      0 [::]:ldaps              [::]:*                  LISTEN
tcp6       0      0 [::]:submission         [::]:*                  LISTEN

Also didn't notice the wallace error that https://git.kolab.org/T7247#102107 and https://git.kolab.org/T7247#100099 mentioned, though I'm not sure where that should appear (there's no wallace log file in /var/log AFAICT).

Additional logs

Guam crashing

During testing Guam was crashing (every 10 seconds, probably a process starting/calling it). Some log output in case it is relevant later on:

root@kolab:/var/log/guam# tail crash.log -n50
    reductions: 17278
  neighbours:
2023-06-01 04:16:07 =ERROR REPORT====
** State machine <0.2327.0> terminating
** Last event in was {connect,<0.2243.0>,server_hello}
** When State == disconnected
**      Data  == {state,"127.0.0.1",9993,true,false,undefined,<<>>,1,{[],[]},undefined,undefined,false,undefined,<<>>,undefined,undefined,false,infinity,none}
** Reason for termination ==
** {{badmatch,{{error,closed},true,<0.2327.0>,{[],[]}}},[{eimap,disconnected,2,[{file,"/usr/src/packages/BUILD/_checkouts/eimap/src/eimap.erl"},{line,140}]},{gen_fsm,handle_msg,8,[{file,"gen_fsm.erl"},{line,479}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
2023-06-01 04:16:07 =CRASH REPORT====
  crasher:
    initial call: eimap:init/1
    pid: <0.2327.0>
    registered_name: []
    exception exit: {{{badmatch,{{error,closed},true,<0.2327.0>,{[],[]}}},[{eimap,disconnected,2,[{file,"/usr/src/packages/BUILD/_checkouts/eimap/src/eimap.erl"},{line,140}]},{gen_fsm,handle_msg,8,[{file,"gen_fsm.erl"},{line,479}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},[{gen_fsm,terminate,8,[{file,"gen_fsm.erl"},{line,607}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
    ancestors: [<0.2243.0>,<0.1929.0>,kolab_guam_sup,<0.1924.0>]
    message_queue_len: 0
    messages: []
    links: [<0.2243.0>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 4185
    stack_size: 28
    reductions: 17278
  neighbours:
2023-06-01 04:16:17 =ERROR REPORT====
** State machine <0.2335.0> terminating
** Last event in was {connect,<0.2251.0>,server_hello}
** When State == disconnected
**      Data  == {state,"127.0.0.1",9993,true,false,undefined,<<>>,1,{[],[]},undefined,undefined,false,undefined,<<>>,undefined,undefined,false,infinity,none}
** Reason for termination ==
** {{badmatch,{{error,closed},true,<0.2335.0>,{[],[]}}},[{eimap,disconnected,2,[{file,"/usr/src/packages/BUILD/_checkouts/eimap/src/eimap.erl"},{line,140}]},{gen_fsm,handle_msg,8,[{file,"gen_fsm.erl"},{line,479}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
2023-06-01 04:16:17 =CRASH REPORT====
  crasher:
    initial call: eimap:init/1
    pid: <0.2335.0>
    registered_name: []
    exception exit: {{{badmatch,{{error,closed},true,<0.2335.0>,{[],[]}}},[{eimap,disconnected,2,[{file,"/usr/src/packages/BUILD/_checkouts/eimap/src/eimap.erl"},{line,140}]},{gen_fsm,handle_msg,8,[{file,"gen_fsm.erl"},{line,479}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},[{gen_fsm,terminate,8,[{file,"gen_fsm.erl"},{line,607}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
    ancestors: [<0.2251.0>,<0.1929.0>,kolab_guam_sup,<0.1924.0>]
    message_queue_len: 0
    messages: []
    links: [<0.2251.0>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 4185
    stack_size: 28
    reductions: 17278
  neighbours:
root@kolab:/var/log/guam# tail erlang.log.1
04:22:48.744 [error] gen_fsm <0.2657.0> in state disconnected terminated with reason: no match of right hand value {{error,closed},true,<0.2657.0>,{[],[]}} in eimap:disconnected/2 line 140
04:22:48.745 [error] CRASH REPORT Process <0.2657.0> with 0 neighbours exited with reason: no match of right hand value {{error,closed},true,<0.2657.0>,{[],[]}} in eimap:disconnected/2 line 140 in gen_fsm:terminate/8 line 607
04:22:58.774 [error] gen_fsm <0.2666.0> in state disconnected terminated with reason: no match of right hand value {{error,closed},true,<0.2666.0>,{[],[]}} in eimap:disconnected/2 line 140
04:22:58.775 [error] CRASH REPORT Process <0.2666.0> with 0 neighbours exited with reason: no match of right hand value {{error,closed},true,<0.2666.0>,{[],[]}} in eimap:disconnected/2 line 140 in gen_fsm:terminate/8 line 607
04:23:08.804 [error] gen_fsm <0.2674.0> in state disconnected terminated with reason: no match of right hand value {{error,closed},true,<0.2674.0>,{[],[]}} in eimap:disconnected/2 line 140
04:23:08.804 [error] CRASH REPORT Process <0.2674.0> with 0 neighbours exited with reason: no match of right hand value {{error,closed},true,<0.2674.0>,{[],[]}} in eimap:disconnected/2 line 140 in gen_fsm:terminate/8 line 607
04:23:18.838 [error] gen_fsm <0.2682.0> in state disconnected terminated with reason: no match of right hand value {{error,closed},true,<0.2682.0>,{[],[]}} in eimap:disconnected/2 line 140
04:23:18.838 [error] CRASH REPORT Process <0.2682.0> with 0 neighbours exited with reason: no match of right hand value {{error,closed},true,<0.2682.0>,{[],[]}} in eimap:disconnected/2 line 140 in gen_fsm:terminate/8 line 607
04:23:28.869 [error] gen_fsm <0.2690.0> in state disconnected terminated with reason: no match of right hand value {{error,closed},true,<0.2690.0>,{[],[]}} in eimap:disconnected/2 line 140
04:23:28.869 [error] CRASH REPORT Process <0.2690.0> with 0 neighbours exited with reason: no match of right hand value {{error,closed},true,<0.2690.0>,{[],[]}} in eimap:disconnected/2 line 140 in gen_fsm:terminate/8 line 607

Probably better logs:

root@kolab:/var/log/guam/log# tail error.log
2023-06-01 04:32:41.439 [error] <0.3154.0>@eimap:disconnected:140 gen_fsm <0.3154.0> in state disconnected terminated with reason: no match of right hand value {{error,closed},true,<0.3154.0>,{[],[]}} in eimap:disconnected/2 line 140
2023-06-01 04:32:41.439 [error] <0.3154.0>@eimap:disconnected:140 CRASH REPORT Process <0.3154.0> with 0 neighbours exited with reason: no match of right hand value {{error,closed},true,<0.3154.0>,{[],[]}} in eimap:disconnected/2 line 140 in gen_fsm:terminate/8 line 607
2023-06-01 04:32:51.465 [error] <0.3162.0>@eimap:disconnected:140 gen_fsm <0.3162.0> in state disconnected terminated with reason: no match of right hand value {{error,closed},true,<0.3162.0>,{[],[]}} in eimap:disconnected/2 line 140
2023-06-01 04:32:51.465 [error] <0.3162.0>@eimap:disconnected:140 CRASH REPORT Process <0.3162.0> with 0 neighbours exited with reason: no match of right hand value {{error,closed},true,<0.3162.0>,{[],[]}} in eimap:disconnected/2 line 140 in gen_fsm:terminate/8 line 607
2023-06-01 04:33:01.501 [error] <0.3171.0>@eimap:disconnected:140 gen_fsm <0.3171.0> in state disconnected terminated with reason: no match of right hand value {{error,closed},true,<0.3171.0>,{[],[]}} in eimap:disconnected/2 line 140
2023-06-01 04:33:01.501 [error] <0.3171.0>@eimap:disconnected:140 CRASH REPORT Process <0.3171.0> with 0 neighbours exited with reason: no match of right hand value {{error,closed},true,<0.3171.0>,{[],[]}} in eimap:disconnected/2 line 140 in gen_fsm:terminate/8 line 607
2023-06-01 04:33:11.533 [error] <0.3179.0>@eimap:disconnected:140 gen_fsm <0.3179.0> in state disconnected terminated with reason: no match of right hand value {{error,closed},true,<0.3179.0>,{[],[]}} in eimap:disconnected/2 line 140
2023-06-01 04:33:11.533 [error] <0.3179.0>@eimap:disconnected:140 CRASH REPORT Process <0.3179.0> with 0 neighbours exited with reason: no match of right hand value {{error,closed},true,<0.3179.0>,{[],[]}} in eimap:disconnected/2 line 140 in gen_fsm:terminate/8 line 607
2023-06-01 04:33:21.560 [error] <0.3187.0>@eimap:disconnected:140 gen_fsm <0.3187.0> in state disconnected terminated with reason: no match of right hand value {{error,closed},true,<0.3187.0>,{[],[]}} in eimap:disconnected/2 line 140
2023-06-01 04:33:21.560 [error] <0.3187.0>@eimap:disconnected:140 CRASH REPORT Process <0.3187.0> with 0 neighbours exited with reason: no match of right hand value {{error,closed},true,<0.3187.0>,{[],[]}} in eimap:disconnected/2 line 140 in gen_fsm:terminate/8 line 607
root@kolab:/var/log/guam/log# tail console.log
2023-06-01 04:32:41.439 [error] <0.3154.0>@eimap:disconnected:140 gen_fsm <0.3154.0> in state disconnected terminated with reason: no match of right hand value {{error,closed},true,<0.3154.0>,{[],[]}} in eimap:disconnected/2 line 140
2023-06-01 04:32:41.439 [error] <0.3154.0>@eimap:disconnected:140 CRASH REPORT Process <0.3154.0> with 0 neighbours exited with reason: no match of right hand value {{error,closed},true,<0.3154.0>,{[],[]}} in eimap:disconnected/2 line 140 in gen_fsm:terminate/8 line 607
2023-06-01 04:32:51.465 [error] <0.3162.0>@eimap:disconnected:140 gen_fsm <0.3162.0> in state disconnected terminated with reason: no match of right hand value {{error,closed},true,<0.3162.0>,{[],[]}} in eimap:disconnected/2 line 140
2023-06-01 04:32:51.465 [error] <0.3162.0>@eimap:disconnected:140 CRASH REPORT Process <0.3162.0> with 0 neighbours exited with reason: no match of right hand value {{error,closed},true,<0.3162.0>,{[],[]}} in eimap:disconnected/2 line 140 in gen_fsm:terminate/8 line 607
2023-06-01 04:33:01.501 [error] <0.3171.0>@eimap:disconnected:140 gen_fsm <0.3171.0> in state disconnected terminated with reason: no match of right hand value {{error,closed},true,<0.3171.0>,{[],[]}} in eimap:disconnected/2 line 140
2023-06-01 04:33:01.501 [error] <0.3171.0>@eimap:disconnected:140 CRASH REPORT Process <0.3171.0> with 0 neighbours exited with reason: no match of right hand value {{error,closed},true,<0.3171.0>,{[],[]}} in eimap:disconnected/2 line 140 in gen_fsm:terminate/8 line 607
2023-06-01 04:33:11.533 [error] <0.3179.0>@eimap:disconnected:140 gen_fsm <0.3179.0> in state disconnected terminated with reason: no match of right hand value {{error,closed},true,<0.3179.0>,{[],[]}} in eimap:disconnected/2 line 140
2023-06-01 04:33:11.533 [error] <0.3179.0>@eimap:disconnected:140 CRASH REPORT Process <0.3179.0> with 0 neighbours exited with reason: no match of right hand value {{error,closed},true,<0.3179.0>,{[],[]}} in eimap:disconnected/2 line 140 in gen_fsm:terminate/8 line 607
2023-06-01 04:33:21.560 [error] <0.3187.0>@eimap:disconnected:140 gen_fsm <0.3187.0> in state disconnected terminated with reason: no match of right hand value {{error,closed},true,<0.3187.0>,{[],[]}} in eimap:disconnected/2 line 140
2023-06-01 04:33:21.560 [error] <0.3187.0>@eimap:disconnected:140 CRASH REPORT Process <0.3187.0> with 0 neighbours exited with reason: no match of right hand value {{error,closed},true,<0.3187.0>,{[],[]}} in eimap:disconnected/2 line 140 in gen_fsm:terminate/8 line 607
root@kolab:/var/log/guam/log#
sevens added a comment.Jun 6 2023, 12:02 PM

Some quick further testing.

TL;DR:

  • Mail and calendar data via Kontact seems to work, at least somewhat
  • Calendar via roundcube is broken, same issue as @tilmankroesche (T7247#100090). Trying Git doesn't help, investigated a bunch more, (hopefully) ruled out some causes but no solution.
  • roundcubemail is a single self-contained package, containing Roundcube and all the plugins (including all of Kolab's that were separate packages before). The other roundcubemail-* packages are old. At least roundcubemail-plugins-kolab still gets installed but it's nearly empty (2 small files and some empty dir hierarchies). roundcube-* packages seem to be from Debian proper(?) and not relevant.
  • roundcubemail from "Testing" repo already contains the Git version of (what was separately in) roundcubemail-plugins-kolab, roundcubemail-plugin-libkolab, etc.
  • Cause of the SWIG\_p_Kolab__cDateTime::isValid() error seems to be outside of roundcubemail-plugins-kolab (possibly in libkolabxml, php-kolabformat (its PHP bindings) and/or Swig (generated these bindings)). Reached limits of my C/Swig knowledge, didn't check further.

Testing Kolab via Kontact

Added Kolab to Kontact on a test setup so I can do some testing with that

Mail:

  • The 'Sent' mails I can see in roundcube (and can still see on new login) are visible in kmail too
  • As a quick test added a mail to the 'Draft' folder from KMail. Seems stored and is visible in Roundcube as well

Calendar:

  • Items added via Roundcube's web interface are visible in KOrganizer
    • NOTE: the items, regardless of which client created them, aren't visible in the Roundcube interface itself, at least not after a few seconds/minutes or a refresh; see below
  • Items created via KOrganizer are visible in KOrganizer
  • Restarting Kontact for second time made the KOrganizer items disappear, possibly a local bug
    • Some more details, in case they're relevant:
      • In Akonadi Console it still shows the 5 items I created in total (both as "unread" and "total") in the collection listing but no individual items (in the collection's item listing) after selecting it. Resyncing and even a reboot didn't help.
      • Creating 3 more items in Korganizer, after a reboot, does show the 3 new items in both Korganizer and Akonadi console. Latter shows a total of 8, the previous 5 are still invisible/inaccessible, the 3 new ones are visible and accessible
      • A few of the 5 missing items *are* still visible in KOrganizer, under 'Open Invitations', showing them as belonging to the Kolab resource's calendar. Interestingly the 3 newly created ones aren't visible under this.
      • Adding more items and syncing more makes it even more of and a weirder mess.

Calendar via Roundcube/SWIG\_p_Kolab__cDateTime::isValid() error

Issue found during testing:

Investigating

Curl of the request (to replicate):

$  curl 'http://kolab.example.com/roundcubemail/QO8P6xKqpHrFmQ2R/?_task=calendar&source=Q2FsZW5kYXI&_action=load_events&start=2023-06-05T00%3A00%3A00&end=2023-06-12T00%3A00%3A00&_=1686022298480' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113.0' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Accept-Language: en-US' -H 'Accept-Encoding: gzip, deflate' -H 'Referer: http://kolab.example.com/roundcubemail/QO8P6xKqpHrFmQ2R/?_task=calendar&source=Q2FsZW5kYXI' -H 'X-Roundcube-Request: YXnm8eoGWtaywTjlnL2ApX1RACEiM9wg' -H 'X-Requested-With: XMLHttpRequest' -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Cookie: PHPSESSID=b95q7nkl5hoi6es9f0mev7i89q; roundcube_sessid=liqtf7h5kb4gjpfntcbllqiquo; roundcube_sessauth=aYj50P4IdzIosY5h19Ag3k4rSl-1686020400' -H 'Sec-GPC: 1'

Check log:

root@kolab:/var/log/roundcubemail# tail -f errors.log
... do curl in other terminal ...
[06-Jun-2023 05:37:57 Europe/Amsterdam] PHP Fatal error:  Uncaught Error: Call to undefined method SWIG\_p_Kolab__cDateTime::isValid() in /usr/share/roundcubemail/plugins/libkolab/lib/kolab_format.php:279
Stack trace:
#0 /usr/share/roundcubemail/plugins/libkolab/lib/kolab_format.php(594): kolab_format::php_datetime()
#1 /usr/share/roundcubemail/plugins/libkolab/lib/kolab_format_xcal.php(110): kolab_format->to_array()
#2 /usr/share/roundcubemail/plugins/libkolab/lib/kolab_format_event.php(152): kolab_format_xcal->to_array()
#3 /usr/share/roundcubemail/plugins/libkolab/lib/kolab_storage_folder.php(564): kolab_format_event->to_array()
#4 /usr/share/roundcubemail/plugins/libkolab/lib/kolab_storage_cache.php(1115): kolab_storage_folder->read_object()
#5 /usr/share/roundcubemail/plugins/libkolab/lib/kolab_storage_cache.php(808): kolab_storage_cache->_unserialize()
#6 /usr/share/roundcubemail/plugins/libkolab/lib/kolab_storage_folder.php(321): kolab_storage_cache->select()
#7 /usr/share/roundcubemail/plugins/calendar/drivers/kolab/kolab_calendar.php(319): kolab_storage_fol in /usr/share/roundcubemail/plugins/libkolab/lib/kolab_format.php on line 279

Same error as @tilmankroesche (T7247#100090). ghane (T7247#100105) recommends using the Git checkout. Check versions etc:

root@kolab:~# apt show roundcubemail-plugin-libkolab
Package: roundcubemail-plugin-libkolab
Version: 1:3.5.11-0~kolab1
...
APT-Sources: http://obs.kolabsys.com/repositories/Kolab:/16:/Testing/Debian_11.0 ./ Packages
...

Current release in git is 3.5.11 ( https://git.kolab.org/diffusion/RPK/tags/master/plugins/libkolab/ , tagged 2022-01-14).

Kolab's debian repositories provide:

root@kolab:~# apt-cache madison roundcubemail-plugins-kolab
roundcubemail-plugins-kolab | 1:3.5.11-0~kolab1 | http://obs.kolabsys.com/repositories/Kolab:/16:/Testing/Debian_11.0 ./ Packages

Build custom git package:

  • I tried building a Debian package using the current Git release as source.
  • Tried getting the roundcubemail-plugins-kolab Debian package source and bumping it (against a tarball made from the Git repository's latest commit)
  • The created roundcubemail-plugins-kolab package was pretty much empty (few empty nested dirs and 2 files)
  • Lots of other roundcubemail-plugin-* (note the singular 'plugin'!) packages where created
  • On my current system, of all of these only roundcubemail-plugins-kolab seems to be installed. Unclear what I should do with the others...
  • Some old snapshots I checked (oldest is one I made just after the base was working, during my initial attempts) had the same
  • Couldn't find out which package supplied the roundcubemail/plugins/libkolab/lib/kolab_format.php file to begin with (to know which one to definitely replace with the 'git' version); none reported to own the file
  • There are roundcube, roundcube-core, roundcubemail-core and roundcubemail packages, roundcubemail being at 1.5.4 and the other 3 at 1.4.13
    • roundcube-* seems to be from Debian proper, roundcubemail-* from Kolab
    • Most of the other roundcubemail-* packages are at 1.4.13 (except for Kolab's own and 3rd-party plugin packages)
    • Most of these weren't installed on my system, though roundcubemail-plugins-kolab was
  • Long story short, after a lot of confusion and a bunch of googling (technically DuckDuckGoing(?)), testing and investigating:
    • Seems that most of these packages are deprecated/now integrated in a single self-contained roundcubemail package. Both old and new are still available in the same repositories though.
    • The version of this roundcubemail package from the 'Testing' repo (1:1.5.4-1~kolab7) already contains the Git version of the old roundcubemail-plugins-kolab, roundcubemail-plugin-libkolab (see bottom section for references)
    • Checking the actual installed code against the latest Git change for the file causing the error (plugins/libkolab/lib/kolab_storage.php) confirms the same version seems to be installed, i.e. the installed package contains the current Git version (2nd @param says bool in new version, boolean in old):
root@kolab:/usr/share/roundcubemail# sed -n 170,180p /usr/share/roundcubemail/plugins/libkolab/lib/kolab_storage.php
     *
     * @param string Data type to list folders for (contact,distribution-list,event,task,note)
     * @param bool   Enable to return subscribed folders only (null to use configured subscription mode)
     *
     * @return array List of Kolab_Folder objects (folder names in UTF7-IMAP)
     */
    public static function get_folders($type, $subscribed = null)
    {
        $folders = $folderdata = array();

        if (self::setup()) {

Tested creating calendar items via Roundcube again as I've done a lot in the process: still same results, same error...

Digging deeper

Checking the source code (kolab_format.php L279 ) the failing object should be a "libkolabxml datetime object"

Checking libkolabxml, seems latest version is installed:

root@kolab:/var/log/roundcubemail# apt search libkolabxml
Sorting... Done
Full Text Search... Done
libkolabxml-dev/unknown 1.3.0-0~kolab2 amd64
  Development files for libkolabxml

libkolabxml1v5/unknown,now 1.3.0-0~kolab2 amd64 [installed,automatic]
  Libkolabxml is the reference implementation of the Kolab XML format

libkolabxml1v5-dbgsym/unknown 1.3.0-0~kolab2 amd64
  debug symbols for libkolabxml1v5

php-kolabformat/unknown,now 1.3.0-0~kolab2 amd64 [installed,automatic]
  PHP bindings for Kolab Format 3.0 XML parsing

python-kolabformat/unknown 1.3.0-0~kolab2 amd64
  Python bindings for Kolab Format 3.0 XML parsing

python3-kolabformat/unknown,now 1.3.0-0~kolab2 amd64 [installed,auto-removable]
  Python 3 bindings for Kolab Format 3.0 XML parsing

root@kolab:/var/log/roundcubemail# apt show libkolabxml1v5
Package: libkolabxml1v5
Version: 1.3.0-0~kolab2
...
N: There are 3 additional records. Please use the '-a' switch to see them.
root@kolab:/var/log/roundcubemail# apt-cache madison libkolabxml1v5
libkolabxml1v5 | 1.3.0-0~kolab2 | http://obs.kolabsys.com/repositories/Kolab:/16/Debian_11.0 ./ Packages
libkolabxml1v5 | 1.3.0-0~kolab2 | http://obs.kolabsys.com/repositories/Kolab:/16:/Testing/Debian_11.0 ./ Packages
libkolabxml1v5 | 1.2.1-0~kolab3 | http://obs.kolabsys.com/repositories/Kolab:/Winterfell/Debian_11.0 ./ Packages
libkolabxml1v5 |    1.2.0-2 | http://deb.devuan.org/merged chimaera/main amd64 Packages
libkolabxml | 1.3.0-0~kolab2 | http://obs.kolabsys.com/repositories/Kolab:/16/Debian_11.0 ./ Sources
libkolabxml | 1.3.0-0~kolab2 | http://obs.kolabsys.com/repositories/Kolab:/16:/Testing/Debian_11.0 ./ Sources
root@kolab:/var/log/roundcubemail# apt-cache madison php-kolabformat
php-kolabformat | 1.3.0-0~kolab2 | http://obs.kolabsys.com/repositories/Kolab:/16/Debian_11.0 ./ Packages
php-kolabformat | 1.3.0-0~kolab2 | http://obs.kolabsys.com/repositories/Kolab:/16:/Testing/Debian_11.0 ./ Packages
php-kolabformat | 1.2.1-0~kolab3 | http://obs.kolabsys.com/repositories/Kolab:/Winterfell/Debian_11.0 ./ Packages
libkolabxml | 1.3.0-0~kolab2 | http://obs.kolabsys.com/repositories/Kolab:/16/Debian_11.0 ./ Sources
libkolabxml | 1.3.0-0~kolab2 | http://obs.kolabsys.com/repositories/Kolab:/16:/Testing/Debian_11.0 ./ Sources

[[ https://git.kolab.org/diffusion/LKX/browse/master/src/?grep=isValid | Searching C source for isValid]] gives lots of results, some in possibly relevant places (NOTE: my C knowledge is quite bare, I can read it a bit and write barely more than a hello world). E.g.: kolabcontainers.cpp L184.

Latest commit to the libkolabxml Git repo seems to be from 2022-08-03. Has 3 commits since last proper tag (1.2.1, so I guess the '1.3.0' pkgs we have are Git master's already), none seem relevant.

Offending line in the code (kolab_format.php L279) was last changed in [[ https://git.kolab.org/rRPK83ac298cd74fb8d30e4aed6146324405e1133a3f | 83ac298cd74f ]] which was commited on 2018-12-13:

  • This change just added explicit {} wrapping so the actual code was in there even before that commit
  • This leads me to believe the problem is *not* in the roundcubemail-plugins-kolab/libkolab code; if so it would've been likely many others would have hit it in the past ~4.5 years
  • NOTE: there have been plenty releases since this commit, just checking the linear history the first tag since that date is for version 3.4.2 (2019-01-19); I assume that included the commit

So problem is then probably either in libkolabxml (either the libkolabxml1v5 or php-kolabformat package) or in Swig itself/how it's configured to be used when building these packages, or a combination thereof.

Tested a downgrade of the libkolabxml packages (hopefuly built with earlier Swig version/config):

root@kolab:/usr/share/roundcubemail# apt install libkolabxml1v5=1.2.1-0~kolab3 php-kolabformat=1.2.1-0~kolab3 python3-kolabformat=1.2.1-0~kolab3
...
root@kolab:/usr/share/roundcubemail# service apache2 restart

This resulted in Apache2 segfaulting so I reverted it.

I don't know what to test/look for further on this issue, C bindings and SWIG are a bit too much above my current level of knowledge

References for the "Testing" package including the Git version for the Kolab plugins
  • https://obs.kolabsys.com/request/show/3014 - most recent OBS change for roundcubemail
    • This indicates roundcubemail-core is old/deprecated and roundcubemail is self-contained.
      • Also notes that the upgrade path for Debian is broken atm ;)
  • Source in testing package:
    • Even the Kolab plugins seem to already be in the source tarball:
root@kolab:~/src# mkdir roundcubemail && cd roundcubemail
root@kolab:~/src/roundcubemail# apt source roundcubemail
...
root@kolab:~/src/roundcubemail# cd roundcubemail-1.5.4/
root@kolab:~/src/roundcubemail/roundcubemail-1.5.4# tar tvf ../roundcubemail_1.5.4.orig.tar.gz | grep -i plugin | grep -i kolab | wc -l
979
root@kolab:~/src/roundcubemail/roundcubemail-1.5.4# apt list --installed roundcubemail
Listing... Done
roundcubemail/unknown,now 1:1.5.4-1~kolab7 all [installed]
and code from latest commit is in this as well:
root@kolab:~/src/roundcubemail/roundcubemail-1.5.4# sed -n 170,180p plugins/libkolab/lib/kolab_storage.php
     *
     * @param string Data type to list folders for (contact,distribution-list,event,task,note)
     * @param bool   Enable to return subscribed folders only (null to use configured subscription mode)
     *
     * @return array List of Kolab_Folder objects (folder names in UTF7-IMAP)
     */
    public static function get_folders($type, $subscribed = null)
    {
        $folders = $folderdata = array();

        if (self::setup()) {
sevens added a comment.Jun 6 2023, 12:23 PM
In T7247#102317, @sevens wrote:

...

Test sending mail

Tested:

  • ...
  • Sent a mail to my personal address (on personal domain)

Results:

  • ...
  • The test mail sent to my personal address didn't arrive either, but I didn't expect it to

...

Well I just checked my spam box and the e-mail *did* arrive, eventually! However, it was sent on 2023-06-01 08:13 and it arrived just this morning on 08:48 (both times copied from webmail interface so should be same timezone)... Just sent another mail to myself and this one arrived immediately. Could very well be on my mail provider's side though.

Times from the headers from first mail:

Received: from mx.soverin.net ([...])
    by storage.soverin.net with LMTP
    id ...
    (envelope-from <peter.sevens@test.example.com>)
    for <peter@DOMAIN.nl>; Tue, 06 Jun 2023 06:48:18 +0000
Received: from kolab.localdomain (XXX-YYY-ZZZ-AAA.MYISP.nl [XXX.YYY.ZZZ.AAA])
    (using ...)
    (No client certificate requested)
    by mx.soverin.net (Postfix) with ESMTPS id ...
    for <peter@DOMAIN.nl>; Tue, 6 Jun 2023 06:48:18 +0000 (UTC)
Authentication-Results: mx.soverin.net;
    dkim=none;
    dmarc=none;
    spf=none (mx.soverin.net: domain of peter.sevens@test.example.com has no SPF policy when checking XXX.YYY.ZZZ.AAA) smtp.mailfrom=peter.sevens@test.example.com
...
Date: Thu, 01 Jun 2023 08:13:52 +0200
...
ghane mentioned this in T7598: Set up Debian 11 with kolab 16.Jul 28 2023, 12:03 PM
ghane added a subtask: T7508: Missing roundcubemail-plugins-kolab package in debian 11 repository.Jul 28 2023, 12:06 PM
mollekopf closed subtask T7508: Missing roundcubemail-plugins-kolab package in debian 11 repository as Resolved.Aug 10 2023, 9:23 AM
tasis mentioned this in T7502: guam 0.9.13-1 is crashing.Aug 12 2023, 6:19 AM
astark subscribed.Aug 16 2023, 11:58 AM
machniak added a project: Kolab 16.Sep 27 2023, 12:26 PM
tasis subscribed.Oct 15 2023, 9:55 AM
sicherha added a comment.Jan 7 2024, 6:02 PM

The broken install dependency kolab-webclient -> roundcubemail-plugins-kolab should be fixed by https://obs.kolabsys.com/request/show/3262, which is currently under review.

For those who are audacious and/or impatient, try adding the following line to /etc/apt/sources.list.d/kolab.list to get the packages built from my personal repo where I'm trying to sort out the Debian 11 issues:

deb https://obs.kolabsys.com/repositories/home:/sicherha:/branches:/Kolab:/16/Debian_11.0/ ./
ghane mentioned this in T8379: Upgrade path from Ubuntu 18.04 or the future of kolab packages.Jul 22 2024, 4:08 PM