Out of memory: Guam dropped 17823 messages in the last second
Open, HighPublic

Description

guam-0.8.3-1.4.el7.kolab_16.x86_64 throws frequently the following error. There can be only one active client at that time, a K-9 Mail on Android with imap IDLE connections.

guam: 00:29:03.000 [warning] lager_error_logger_h dropped 17823 messages in the last second that exceeded the limit of 50 messages/sec

And sometimes this follows:

kernel: Out of memory: Kill process 2492 (beam) score 333 or sacrifice child

guam crash.log entry (there are a lot of them):

2016-11-15 00:43:13 =ERROR REPORT====
** Generic server <0.18831.84> terminating
** Last message in was {'$gen_cast',accept}
** When Server state == {state,{sslsocket,nil,{#Port<0.1388>,{config,{ssl_options,tls,[{3,3},{3,2},{3,1}],verify_none,{#Fun<ssl.7.104555794>,[]},#Fun<ssl.8.104555794>,false,false,undefined,1,<<"/etc/pki/cyrus-imapd/cyrus-imapd.pem">>,undefined,<<"/etc/pki/cyrus-imapd/cyrus-imapd.pem">>,undefined,[],undefined,<<>>,u
ndefined,undefined,undefined,undefined,undefined,[<<"À,">>,<<"À0">>,<<"À$">>,<<"À(">>,<<"À.">>,<<"À2">>,<<"À&">>,<<"À*">>,<<0,159>>,<<0,163>>,<<0,107>>,<<0,106>>,<<0,157>>,<<0,61>>,<<"À+">>,<<"À/">>,<<"À#">>,<<"À'">>,<<"À-">>,<<"À1">>,<<"À%">>,<<"À)">>,<<0,158>>,<<0,162>>,<<0,103>>,<<0,64>>,<<0,156>>,<<0,60>>,<<"À\
n">>,<<192,20>>,<<0,57>>,<<0,56>>,<<192,5>>,<<192,15>>,<<0,53>>,<<"À\b">>,<<192,18>>,<<0,22>>,<<0,19>>,<<192,3>>,<<"À\r">>,<<0,10>>,<<"À\t">>,<<192,19>>,<<0,51>>,<<0,50>>,<<192,4>>,<<192,14>>,<<0,47>>,<<0,21>>,<<0,9>>],#Fun<ssl.1.104555794>,true,268435456,false,true,undefined,false,undefined,undefined,undefined,und
efined,true,undefined,[],undefined,false,true,undefined,false,{ssl_crl_cache,{internal,[]}},[{sha512,ecdsa},{sha512,rsa},{sha384,ecdsa},{sha384,rsa},{sha256,ecdsa},{sha256,rsa},{sha224,ecdsa},{sha224,rsa},{sha,ecdsa},{sha,rsa},{sha,dsa},{md5,rsa}]},[{reuseaddr,true},{active,false},inet6],<0.106.0>,[inet6,{reuseaddr
,true},{packet_size,0},{packet,0},{header,0},{active,false},{mode,binary}],{gen_tcp,tcp,tcp_closed,tcp_error},tls_connection}}},undefined,<0.94.0>,[{certfile,"/etc/pki/cyrus-imapd/cyrus-imapd.pem"}],true,false,[{host,"127.0.0.1"},{port,9993},{tls,true}],[],[{kolab_guam_rule_filter_groupware,{state,undefined,<<>>,fa
lse,<<>>,[<<"LIST">>,<<"list">>,<<"XLIST">>,<<"xlist">>,<<"LSUB">>,<<"lsub">>]}}],undefined,undefined,undefined}
** Reason for termination ==
** {{badmatch,{error,emfile}},[{kolab_guam_session,post_accept_bookkeeping,4,[{file,"src/kolab_guam_session.erl"},{line,143}]},{kolab_guam_session,accept_client,1,[{file,"src/kolab_guam_session.erl"},{line,127}]},{kolab_guam_session,handle_cast,2,[{file,"src/kolab_guam_session.erl"},{line,51}]},{gen_server,try_disp
atch,4,[{file,"gen_server.erl"},{line,615}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,681}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}
2016-11-15 00:43:13 =CRASH REPORT====
  crasher:
    initial call: kolab_guam_session:init/1
    pid: <0.18831.84>
    registered_name: []
    exception exit: {{{badmatch,{error,emfile}},[{kolab_guam_session,post_accept_bookkeeping,4,[{file,"src/kolab_guam_session.erl"},{line,143}]},{kolab_guam_session,accept_client,1,[{file,"src/kolab_guam_session.erl"},{line,127}]},{kolab_guam_session,handle_cast,2,[{file,"src/kolab_guam_session.erl"},{line,51}]},{g
en_server,try_dispatch,4,[{file,"gen_server.erl"},{line,615}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,681}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]},[{gen_server,terminate,7,[{file,"gen_server.erl"},{line,826}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}
]}]}
    ancestors: [<0.94.0>,kolab_guam_sup,<0.89.0>]
    messages: []
    links: [<0.94.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 1598
    stack_size: 27
    reductions: 273
  neighbours:

guam console.log:

2016-11-15 00:43:13.441 [error] <0.18831.84> gen_server <0.18831.84> terminated with reason: no match of right hand value {error,emfile} in kolab_guam_session:post_accept_bookkeeping/4 line 143
2016-11-15 00:43:13.441 [error] <0.18831.84> CRASH REPORT Process <0.18831.84> with 0 neighbours exited with reason: no match of right hand value {error,emfile} in kolab_guam_session:post_accept_bookkeeping/4 line 143 in gen_server:terminate/7 line 826

and also:

2016-11-15 01:01:45.309 [error] <0.116.0> gen_server <0.116.0> terminated with reason: no match of right hand value {error,closed} in kolab_guam_session:accept_client/1 line 129

guam-0.8.3-1.5.el7.kolab_16.x86_64 has this problem too.

Thanks in advance! (If the <<"X">> thingy from the crash log contain sensitive info, please remove that part from my report. Thanks!)

Details

Ticket Type
Task
terba created this task.Nov 15 2016, 8:45 AM
terba raised the priority of this task from 60 to High.Nov 15 2016, 8:48 AM
terba added projects: Guam, Restricted Project.
terba updated the task description. (Show Details)Nov 17 2016, 10:37 AM
{badmatch,{error,emfile}

out of file handles.

2016-11-15 01:01:45.309 [error] <0.116.0> gen_server <0.116.0> terminated with reason: no match of right hand value {error,closed} in kolab_guam_session:accept_client/1 line 129

that's fine. the client closed the connection before guam picked up, and so the session terminated. it's a correct error. (sounds odd, i know, but this is how erlang works, and it prevents having to do defensive programming everywhere)

as for the out of memory issues, that's the kernel OOM killer? if so, that may have very little to do with guam itself and just guam getting nominated for destruction because of kernel memory pressure.

seigo moved this task from Backlog to Pending on the Guam board.Dec 12 2016, 2:29 PM
seigo moved this task from Pending to Backlog on the Guam board.Dec 13 2016, 2:01 PM
pasik added a subscriber: pasik.Nov 25 2017, 2:29 PM