Cyrus IMAP Murder frontend proxies GETANNOTATION commands for GETMETADATA requests
Closed, ResolvedPublic13 Story Points
Actions

Assigned To

Authored By

	adomaitis
	Feb 25 2016, 10:59 AM

Description

Cyrus IMAP Murder frontends are given a command similar to the following:

. GETMETADATA (DEPTH infinity) "*" "/shared/*"

They will still (hard-coded) use GETANNOTATION against backends, yielding untagged responses such as:

* ANNOTATION "Calendar" "/vendor/kolab/folder-type" ("value.shared" "event" "shared.size" "5" "value.priv" "event.default" "priv.size" "13")

A configuration item in /etc/imapd.conf of suppress_capabilities: METADATA will prevent client applications from using GETMETADATA and SETMETADATA, and therefore issue GETANNOTATION and SETANNOTATION commands, with untagged responses ANNOTATION.

However, when a client application does issue GETMETADATA, the response in a discrete Murder topology will yield untagged ANNOTATION responses -- that the client application may not parse.

The client application in question here is Guam, which has never been developed against the (now legacy) ANNOTATEMORE RFC draft -- it uses METADATA, even if the capability is suppressed.

In contrast, suppressing the METADATA make Roundcube and KDE PIM use ANNOTATEMORE and ANNOTATEMORE2 (if available).

This results in the following options:

Make Guam use ANNOTATEMORE rather than METADATA, so a discrete Murder topology's frontends can use their (hard-coded) GETANNOTATION and SETANNOTATION,
Change everything over to METADATA and drop ANNOTATEMORE altogether. This would include;
- Changing Cyrus IMAP to use METADATA internally, and
- Working around the lack of METADATA support in Python's imaplib (through cyruslib, shipped as part of PyKolab), updating calls to getannotation() and setannotation() functions used internally by PyKolab, and
- Optionally, dropping support for ANNOTATEMORE and ANNOTATEMORE2 from clients including Roundcube, Roundcube Kolab Plugins and KDE PIM and Kube.

Details

Ticket Type: Task

Related Objects
Search...

Status	Assigned	Task
Resolved	• seigo	T995 Run Guam in Beta
Resolved	vanmeeuwen	T1062 Cyrus IMAP Murder frontend proxies GETANNOTATION commands for GETMETADATA requests
Spite	• seigo	T1108 Prefer ANNOTATEMORE over METADATA

Event Timeline

adomaitis created this task.Feb 25 2016, 10:59 AM

We need a more accurately detailed transcript of the IMAP sessions.

Not sure if that is enough.

imtest -u user.bandymui@domain.com -a user.bandymui@domain.com -t '' -w 'pass' -p 9143 guam.domain.tld

S: * OK [CAPABILITY STARTTLS IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxten QUOTA NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SORT SORT=MODSEQ SORT=DISPLAY SORT=UID THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE ANNOTATE-EXPERIMENT-1 LIST-EXTENDED LIST-STATUS LIST-MYRIGHTS WITHIN QRESYNC SCAN XLIST XMOVE MOVE URLAUTH URLAUTH=BINARY X-NETSCAPE MUPDATE=mupdate://mupdate.domain.tld/ AUTH=PLAIN SASL-IR COMPRESS=DEFLATE X-QUOTA=STORAGE X-QUOTA=MESSAGE X-QUOTA=X-ANNOTATION-STORAGE X-QUOTA=X-NUM-FOLDERS IDLE] pomidoras.domain.tld Cyrus IMAP Murder 2.5.6.7-Kolab-2.5.6-8.1.el6.kolab_14 server ready
C: S01 STARTTLS
S: S01 OK Begin TLS negotiation now
verify error:num=20:unable to get local issuer certificate
verify error:num=27:certificate not trusted
verify error:num=21:unable to verify the first certificate
TLS connection established: TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)
C: C01 CAPABILITY
S: * CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxten QUOTA NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SORT SORT=MODSEQ SORT=DISPLAY SORT=UID THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE ANNOTATE-EXPERIMENT-1 LIST-EXTENDED LIST-STATUS LIST-MYRIGHTS WITHIN QRESYNC SCAN XLIST XMOVE MOVE URLAUTH URLAUTH=BINARY X-NETSCAPE MUPDATE=mupdate://mupdate.domain.tld/ AUTH=PLAIN SASL-IR COMPRESS=DEFLATE X-QUOTA=STORAGE X-QUOTA=MESSAGE X-QUOTA=X-ANNOTATION-STORAGE X-QUOTA=X-NUM-FOLDERS IDLE
S: C01 OK Completed
C: A01 AUTHENTICATE PLAIN W5mb3NhaXRhcy5iYW5keW11aUBtaW5kc2hhcmViYWx0aWNzLmNvbQBpbmZvc2FpdGFzLmJhbmR5bXVpQG1pbmRzaGFyZWJhbHRpY3MuY29tACFhdDVwZWs
S: A01 OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxten QUOTA NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SORT SORT=MODSEQ SORT=DISPLAY SORT=UID THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE ANNOTATE-EXPERIMENT-1 LIST-EXTENDED LIST-STATUS LIST-MYRIGHTS WITHIN QRESYNC SCAN XLIST XMOVE MOVE URLAUTH URLAUTH=BINARY X-NETSCAPE MUPDATE=mupdate://mupdate.domain.tld/ LOGINDISABLED COMPRESS=DEFLATE X-QUOTA=STORAGE X-QUOTA=MESSAGE X-QUOTA=X-ANNOTATION-STORAGE X-QUOTA=X-NUM-FOLDERS IDLE] Success (tls protection) SESSIONID=<pomidoras.domain.tld-27317-1456478292-1-419227042192432141>
Authenticated.
Security strength factor: 256
a1 LIST "" "*"

LIST (\Noinferiors \HasNoChildren) "/" INBOX
LIST (\HasChildren) "/" Archive
LIST (\HasNoChildren) "/" Archive/INBOX
LIST (\HasChildren) "/" Archive/Projektai
LIST (\HasNoChildren) "/" "Archive/Projektai/3 Failai"
LIST (\HasNoChildren) "/" Archive/Sent
LIST (\HasChildren) "/" Calendar
LIST (\HasNoChildren) "/" "Calendar/Personal Calendar"
LIST (\HasNoChildren) "/" Configuration
LIST (\HasChildren) "/" Contacts
LIST (\HasNoChildren) "/" "Contacts/Personal Contacts"
LIST (\HasNoChildren) "/" "Conversation Action Settings"
LIST (\HasNoChildren) "/" "Deleted Items"
LIST (\HasNoChildren) "/" Drafts

....

Does the Guam proxy or the Cyrus IMAP frontend have anything to say on the matter? Guam should be requesting the metadata for each of the folders in the list. Telemetry can tell you if it indeed does.

Investigating where the issue is exactly.

vanmeeuwen edited a custom field.Mar 9 2016, 11:34 AM

vanmeeuwen moved this task from Backlog to Doing on the Sprint 201610 board.

It is the frontend that proxies a GETANNOTATION command to the backend, even though the original command is GETMETADATA.

The backend through answers "correctly", with ANNOTATION responses (to the GETANNOTATION command).

vanmeeuwen renamed this task from guam groupware folder filter doesn't work in Murder cluster setup to Cyrus IMAP Murder frontend proxies GETANNOTATION commands for GETMETADATA requests.Mar 9 2016, 3:48 PM

vanmeeuwen updated the task description. (Show Details)

vanmeeuwen edited a custom field.

vanmeeuwen created subtask T1108: Prefer ANNOTATEMORE over METADATA.Mar 9 2016, 4:16 PM

The fix for this is in https://git.cyrus.foundation/D86

vanmeeuwen closed subtask T1108: Prefer ANNOTATEMORE over METADATA as Spite.Mar 10 2016, 2:56 PM

vanmeeuwen added a parent task: T995: Run Guam in Beta.Mar 10 2016, 3:50 PM

vanmeeuwen mentioned this in T995: Run Guam in Beta.

vanmeeuwen closed this task as Resolved.Mar 10 2016, 3:52 PM

vanmeeuwen moved this task from Review to Done on the Sprint 201610 board.

I can't confirm the issue with guam still displaying groupware folders from Cyrus Murder is fixed.
Detailed description:
I have murder cluster running with cyrus-imapd-2.5.6-8.1. Frontends and Backends have commented suppress_capabilities: METADATA config option.
I have added yet another frontend cyrus-imapd-2.5.7-8.1 to the cluster (from Winterfel repositories). Guam service is running on port 9143 on this new machine as well.

The '*' doesn't work anymore if I talk to 2.5.7 cyrus version.

If I run

echo '. GETMETADATA (DEPTH infinity) "*" (/shared/vendor/kolab/folder-type)' | imtest .... <frontend 2.5.6>
* ANNOTATION INBOX "/vendor/kolab/folder-type" ("value.shared" "mail.inbox")
* ANNOTATION Archive "/vendor/kolab/folder-type" ("value.shared" NIL)
* ANNOTATION Archive/INBOX "/vendor/kolab/folder-type" ("value.shared" NIL)

However running the same echo against frontend 2.5.7 I get:

Protection error: Connection reset by peer

messages log file shows:

kernel: proxyd[17138]: segfault at 7ffffff8 ip 00007f0ef1d5dab4 sp 00007ffca2cf2aa0 error 4 in libc-2.17.so[7f0ef1d15000+1b6000]

Segmentation fault is not a desired behavior I suppose.

Running GETMETADATA and GETANNOTATION on cyrus 2.5.7 frontend returns what it was asked for.

echo '. GETMETADATA (DEPTH infinity) "%" (/shared/vendor/kolab/folder-type)' | imtest ... <frontend 2.5.7> 
* METADATA INBOX (/shared/vendor/kolab/folder-type "mail.inbox")
* METADATA Calendar (/shared/vendor/kolab/folder-type "event.default")

and

echo '. GETANNOTATION "Calendar" "*" ("value.shared" "value.priv")' | imtest ... <frontend 2.5.7>
* ANNOTATION Calendar "/vendor/kolab/color" ("value.shared" "006400")
* ANNOTATION Calendar "/vendor/kolab/folder-type" ("value.shared" "event.default" "value.priv" "event.default")

While the same commands on the frontend 2.5.6 gives me:

echo '. GETMETADATA (DEPTH infinity) "%" (/shared/vendor/kolab/folder-type)' | imtest ... <frontend 2.5.6>
* ANNOTATION INBOX "/vendor/kolab/folder-type" ("value.shared" "mail.inbox")
* ANNOTATION Calendar "/vendor/kolab/folder-type" ("value.shared" "event.default")

echo '. GETANNOTATION "Calendar" "*" ("value.shared" "value.priv")' | imtest .... <frontend 2.5.6>
* ANNOTATION Calendar "/vendor/kolab/color" ("value.shared" "006400")
* ANNOTATION Calendar "/vendor/kolab/folder-type" ("value.shared" "event.default" "value.priv" "event.default")

This is what is expected I suppose. The unpatched Cyrus frontend converts GETMETADATA to GETANNOTATION, while patched Cyrus sends to backend the same command what it got.

Guam is till showing all folders, rather than just mail folders.

Guam is configured to proxy IMAP to frontend running 2.5.7 cyrus version.

echo '. LIST "" "*"' | imtest .... <guam>

doesn't work as explained in 1 issue.

echo '. LIST "" "%"' | imtest .... <guam>
* LIST (\Noinferiors \HasNoChildren) "/" INBOX
* LIST (\HasChildren) "/" Calendar
* LIST (\HasNoChildren) "/" Configuration
* LIST (\HasChildren) "/" Contacts

Telemetry logging on frontend guam is talking to shows nothing different from the output imtest shows, but backend server telemetry shows:

<1457694176<PROXY0 GETMETADATA (DEPTH infinity) "INBOX" ("/shared/vendor/kolab/folder-type" "/shared/vendor/kolab/folder-type/*")
>1457694176>* METADATA INBOX (/shared/vendor/kolab/folder-type "mail.inbox")
PROXY0 OK Completed
<1457694176<N01 NOOP
>1457694176>N01 OK Completed
<1457694176<PROXY1 GETMETADATA (DEPTH infinity) "Archive" ("/shared/vendor/kolab/folder-type" "/shared/vendor/kolab/folder-type/*")
>1457694176>* METADATA Archive (/shared/vendor/kolab/folder-type NIL)
PROXY1 OK Completed
<1457694176<N01 NOOP
>1457694176>N01 OK Completed
<1457694176<PROXY6 GETMETADATA (DEPTH infinity) "Calendar" ("/shared/vendor/kolab/folder-type" "/shared/vendor/kolab/folder-type/*")
>1457694176>* METADATA Calendar (/shared/vendor/kolab/folder-type "event.default")
PROXY6 OK Completed
<1457694176<N01 NOOP
>1457694176>N01 OK Completed
<1457694176<PROXY9 GETMETADATA (DEPTH infinity) "Contacts" ("/shared/vendor/kolab/folder-type" "/shared/vendor/kolab/folder-type/*")
>1457694176>* METADATA Contacts (/shared/vendor/kolab/folder-type "contact.default")
PROXY9 OK Completed
<1457694176<N01 NOOP

This information I read as if guam translates the LIST command to get the METADATA it needs, but nevertheless still shows all folders to the client.

In T1062#16491, @adomaitis wrote:

I can't confirm the issue with guam still displaying groupware folders from Cyrus Murder is fixed.
Detailed description:
I have murder cluster running with cyrus-imapd-2.5.6-8.1. Frontends and Backends have commented suppress_capabilities: METADATA config option.

You should not need to have any suppress_capabilities any longer.

I have added yet another frontend cyrus-imapd-2.5.7-8.1 to the cluster (from Winterfel repositories). Guam service is running on port 9143 on this new machine as well.

The '*' doesn't work anymore if I talk to 2.5.7 cyrus version.

If I run
echo '. GETMETADATA (DEPTH infinity) "*" (/shared/vendor/kolab/folder-type)' | imtest .... <frontend 2.5.6>
* ANNOTATION INBOX "/vendor/kolab/folder-type" ("value.shared" "mail.inbox")
* ANNOTATION Archive "/vendor/kolab/folder-type" ("value.shared" NIL)
* ANNOTATION Archive/INBOX "/vendor/kolab/folder-type" ("value.shared" NIL)
However running the same echo against frontend 2.5.7 I get:
Protection error: Connection reset by peer
messages log file shows:

kernel: proxyd[17138]: segfault at 7ffffff8 ip 00007f0ef1d5dab4 sp 00007ffca2cf2aa0 error 4 in libc-2.17.so[7f0ef1d15000+1b6000]

I cannot reproduce this, but I'm running this on Santiago, and you are running Maipo.

Segmentation fault is not a desired behavior I suppose.

It's not.

Running GETMETADATA and GETANNOTATION on cyrus 2.5.7 frontend returns what it was asked for.

echo '. GETMETADATA (DEPTH infinity) "%" (/shared/vendor/kolab/folder-type)' | imtest ... <frontend 2.5.7> 
* METADATA INBOX (/shared/vendor/kolab/folder-type "mail.inbox")
* METADATA Calendar (/shared/vendor/kolab/folder-type "event.default")

and

echo '. GETANNOTATION "Calendar" "*" ("value.shared" "value.priv")' | imtest ... <frontend 2.5.7>
* ANNOTATION Calendar "/vendor/kolab/color" ("value.shared" "006400")
* ANNOTATION Calendar "/vendor/kolab/folder-type" ("value.shared" "event.default" "value.priv" "event.default")

Guam filters groupware folders from LIST, XLIST and LSUB command responses, but not GETMETADATA, GETANNOTATION.

Please, execute:

. LIST "" "*"

and:

. LIST "" *

The frontend(s) need to run Cyrus IMAP version 2.5.7-8.1. Running any version prior to 2.5.7-8.1 just doesn't contribute to a valid validation scenario.

Guam is till showing all folders, rather than just mail folders.

Guam is configured to proxy IMAP to frontend running 2.5.7 cyrus version.
echo '. LIST "" "*"' | imtest .... <guam>
doesn't work as explained in 1 issue.

You're first issue is about GETMETADATA and GETANNOTATION responses. I'm not sure how this translates to LIST "" "*" not working?

echo '. LIST "" "%"' | imtest .... <guam>
* LIST (\Noinferiors \HasNoChildren) "/" INBOX
* LIST (\HasChildren) "/" Calendar
* LIST (\HasNoChildren) "/" Configuration
* LIST (\HasChildren) "/" Contacts

I don't think the use of the "%" wildcard is treated by Guam at all.

Telemetry logging on frontend guam is talking to shows nothing different from the output imtest shows, (...)

This sounds wrong. When you connect to Guam and issue a LIST command, Guam should trigger the frontend to receive a number of GETMETADATA requests in addition to the LIST command.

In T1062#16492, @vanmeeuwen wrote:
In T1062#16491, @adomaitis wrote:
If I run
. GETMETADATA (DEPTH infinity) "*" (/shared/vendor/kolab/folder-type)
Against 2.5.7:
Protection error: Connection reset by peer
messages log file shows:

kernel: proxyd[17138]: segfault at 7ffffff8 ip 00007f0ef1d5dab4 sp 00007ffca2cf2aa0 error 4 in libc-2.17.so[7f0ef1d15000+1b6000]
I cannot reproduce this, but I'm running this on Santiago, and you are running Maipo.

This does not occur for me on Maipo.

So it looks like the '*' wildcard doesn't work in a mixed version murder environment (backend 2.5.6 and frontend 2.5.7) doesn't work. When all cyrus-imapd instances are of the same version - no more segfaults.

I still can't get guam work with Cyrus Murder, although it is not clear if that depends on if guam is talking to Murder or standalone server. Guam configuration is:

    kolab_guam, [
        {
            imap_servers, [
                {
                    imaps, [
                        { host, "127.0.0.1" },
                        { port, 993 },
                        { tls, true }
                    ]
                }
            ]
        },
        {
            listeners, [
                {
                    imap, [
                        { port, 9143 },
                        { imap_server, imaps },
                        {
                            rules, [
                                { filter_groupware, [] }
                            ]
                        },
                        {
                            tls_config, [
                                { certfile, "/etc/pki/cyrus-imapd/cyrus-imapd.pem" }
                            ]
                        }
                    ]
                },
                {
                    imaps, [
                        { port, 9993 },
                        { implicit_tls, true },
                        { imap_server, imaps },
                        {
                            rules, [
                                { filter_groupware, [] }
                            ]
                        },
                        {
                            tls_config, [
                                { certfile, "/etc/pki/cyrus-imapd/cyrus-imapd.pem" }
                            ]
                        }
                    ]
                }
            ]
        }
    ]
},

issuing command:

# echo '. LIST "" "*"' | imtest -a bandymui@domain.tld -w 'pass' -p 9143 localhost

responds with groupware folders filtered out, however command:

# echo '. LIST "" "*"' | imtest -a bandymui@domain.tld -w 'pass' -t '' -p 9143 localhost

shows all folders, like no filters were applied. The difference in commands is in "-t ''", like talking plain text vs starttls.
I've tried to use hostname IP address instead of 'localhost', works the same way. What is even more strange that if I run the command

# echo '. LIST "" "*"' | imtest -a bandymui@domain.tld -w 'pass' -t '' -p 9143 192.168.17.87

from any other machine, not from the machine where guam is running, I don't see guam filtering any groupware filters. It doesn't matter if I use -t '' or not.
So it looks like guam groupware folders filtering works only if:

connection comes from localhost to localhost
no encryption is used.

Unfortunately guam logs are pretty much useless and I can't find any other way to debug or give you more information.

guam-0.8-7.2.el7.kolab_wf.x86_64
cyrus-imapd-2.5.7-8.1.el7.kolab_wf.x86_64

Cyrus IMAP Murder frontend proxies GETANNOTATION commands for GETMETADATA requestsClosed, ResolvedPublic13 Story PointsActions