/var/log/guam/console.log:2016-08-01 09:24:47.545 [error] <0.28149.319> CRASH REPORT Process <0.28149.319> with 0 neighbours exited with reason: no match of right hand value {error,emfile} in kolab_guam_session:post_accept_bookkeeping/4 line 143 in gen_server:terminate/7 line 826

It is running out of filehandles (as defined in asm-generic/errno-base.h: #define EMFILE 24 /* Too many open files */)

For each connection, at least 2 extra filehandles are required since Guam is a proxy: one for the client to Guam, one for Guam to Cyrus (which gets opened on both sides of that connection, so if Cyrus and Guam are on the same machine, each connection takes a minimum of 3 filehandles). This is of course in addition to whatever normal files are opened by e.g. cyrus to access mail in response to IMAP commands.

Other possibilities that may be complicating matters:

• IMAP clients may not closing their connection to Guam (and so holding open those filehandles), and perhaps even connecting multiple times
• It is possible there is a bug where some filehandle or another is not being closed by Guam. Prior to the 0.8.3 release one such bug was identified and fixed; I have not be able to identify other similar issues, however, having tested the various failure modes (sudden client disconnect, server disconnect, crash/closure of Guam internal connection handler)

So it could be a simple matter of actually running out of file handles due to load. Do we have numbers for how many connections per frontend are active? (Simple lsof | wc -l can be informative in this case ..) As noted in the Guam troubleshooting guide, one may also attach to Guam from an Erlang console, run observer:start() and examine the behavior of the Guam processes in near real time. It would be possible for me to do this remotely even, given appropriate (temporary) access to one of the frontend systems, or via screenshare with a system admin there.

A failure of a load-balancer under the product name of F5 may in fact be creating long-lasting connections as well, as it tends to send a SYN, await the SYN, ACK, but never RST or ACK,FIN.