Avoid duplicates by GID/RID
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	mollekopf
	Apr 15 2016, 9:57 AM

Description

We still get into situations where we have duplicates by GID/RID in the same collection.
We have to:

have a way to resolve this if the situation is already existing (preferably without akonadictl fsck)
Avoid that this can ever happen (in AKAPPEND and MERGE and MOVE).
Figure out what currently triggers this and resolve the root cause (might be easier to find once AKAPPEND/MERGE/MOVE properly fails jobs that trigger the problem).

Details

Ticket Type: Task

Related Objects

Mentioned Here: T793: Kolab resource gets stuck on change replay tasks

Event Timeline

mollekopf created this task.Apr 15 2016, 9:57 AM

mollekopf raised the priority of this task from 40 to 60.Apr 15 2016, 9:59 AM

mollekopf moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.Apr 19 2016, 9:56 AM

See also T793

mollekopf updated the task description. (Show Details)Apr 19 2016, 11:12 AM

When moving a message:

AkonadiConsole Browser Widget (0x7feef8027530) 25 LIST 28 0 (ENABLED TRUE) () 
AkonadiConsole Browser Widget (0x7feef8027530) * 28 13 (NAME "Archive" MIMETYPE (inode/directory message/rfc822) REMOTEID "/Archive" REMOTEREVISION "" RESOURCE "akonadi_kolab_resource_0" VIRTUAL 0 CACHEPOLICY (INHERIT true INTERVAL -1 CACHETIMEOUT -1 SYNCONDEMAND true LOCALPARTS (ENVELOPE HEAD RFC822)) ENABLED TRUE DISPLAY DEFAULT SYNC DEFAULT INDEX DEFAULT AccessRights "a" imapacl "john.doe@example.org lrswipckxtdaen %% %% lrswipckxtdaen" collectionidentification "(\"Archive\" \"\" \"\" \"\" \"\")" collectionannotations "/shared % /shared/check % /shared/checkperiod % /shared/comment % /shared/sort % /shared/specialuse % /shared/thread % /shared/vendor/cmu/cyrus-imapd/duplicatedeliver false % /shared/vendor/cmu/cyrus-imapd/expire % /shared/vendor/cmu/cyrus-imapd/lastpop % /shared/vendor/cmu/cyrus-imapd/news2mail % /shared/vendor/cmu/cyrus-imapd/partition default % /shared/vendor/cmu/cyrus-imapd/pop3newuidl true % /shared/vendor/cmu/cyrus-imapd/pop3showafter % /shared/vendor/cmu/cyrus-imapd/sharedseen false % /shared/vendor/cmu/cyrus-imapd/sieve % /shared/vendor/cmu/cyrus-imapd/size 0 % /shared/vendor/cmu/cyrus-imapd/squat % /shared/vendor/cmu/cyrus-imapd/uniqueid 5baaaf87-6b61-442b-9a11-9cd542f1a3dd % /shared/vendor/horde/share-params % /shared/vendor/kolab/activesync % /shared/vendor/kolab/color % /shared/vendor/kolab/displayname % /shared/vendor/kolab/folder-test % /shared/vendor/kolab/folder-type % /shared/vendor/kolab/h-share-attr-desc % /shared/vendor/kolab/incidences-for % /shared/vendor/kolab/pxfb-readable-for % /shared/vendor/kolab/uniqueid % /shared/vendor/x-toltec/test " collectionquota "0 0" imapquota "Archive %%%% STORAGE % 0 %%%% STORAGE % 0" uidnext "2" collectionflags "\\Answered \\Flagged \\Draft \\Deleted \\Seen \\*" uidvalidity "1456167136" highestmodseq "1") 
AkonadiConsole Browser Widget (0x7feef8027530) 25 OK List completed 
AkonadiConsole Browser Widget (0x7feef8027530) 26 UID MOVE 3065 51 
AkonadiConsole Browser Widget (0x7feef8027530) 26 OK MOVE complete 
akonadi_kolab_resource_0 (0x7fef5c0148b0) 3763 UID FETCH 3065 FULLPAYLOAD CACHEONLY ANCESTORS INF EXTERNALPAYLOAD (UID COLLECTIONID FLAGS SIZE REMOTEID REMOTEREVISION) 
akonadi_kolab_resource_0 (0x7fef5c0148b0) * 3065 FETCH (UID 3065 REV 0 REMOTEID "1" MIMETYPE "message/rfc822" COLLECTIONID 51 SIZE 5545 FLAGS ($ATTACHMENT \SEEN) ANCESTORS ((51 "/sub") (28 "/Archive") (13 "imap://john.doe@example.org@kolab/") (0 "")) PLD:ENVELOPE[1] {227} ("Mon, 18 Apr 2016 16:49:21 +0200" "Booooooooooooooooooooooooooooooo" (("Jane Doe" NIL "doe2" "example.org")) NIL NIL (("John Doe" NIL "john.doe" "example.org")) NIL NIL NIL "<20160418145015.59B5F63E43@kolab1.example.org>" NIL) PLD:HEAD[1] {1031} Return-Path: <doe2@example.org> Received: from kolab1.example.org ([unix socket]) by kolab1.example.org (Cyrus 2.5.2.3-Kolab-2.5.2-5.el7.kolab_3.5) with LMTPA; Mon, 18 Apr 2016 14:50:15 +0000 X-Sieve: CMU Sieve 2.4 X-Virus-Scanned: amavisd-new at example.org X-Spam-Flag: NO X-Spam-Score: 0.242 X-Spam-Level: X-Spam-Status: No, score=0.242 tagged_above=-10 required=6.2 tests=[ALL_TRUSTED=-1, MISSING_MID=0.14, TRACKER_ID=1.102] autolearn=no autolearn_force=no Received: from 205d6d77f0aa.localnet (unknown [172.17.0.4]) by kolab1.example.org (Postfix) with ESMTPSA id 7D6EB63E42 for <john.doe@example.org>; Mon, 18 Apr 2016 14:49:31 +0000 (UTC) User-Agent: KOrganizer/4.13 (Linux/4.4.6-300.fc23.x86_64; KDE/4.14.3; x86_64; ) From: Jane Doe <doe2@example.org> To: John Doe <john.doe@example.org> Date: Mon, 18 Apr 2016 16:49:21 +0200 Subject: Booooooooooooooooooooooooooooooo Content-Type: multipart/mixed; boundary="nextPart4337192.OvIrPZzYpk" MIME-Version: 1.0 Message-Id: <20160418145015.59B5F63E43@kolab1.example.org> PLD:RFC822[1] [FILE] {7} 3210_r0) 
akonadi_kolab_resource_0 (0x7fef5c0148b0) 3763 OK UID FETCH completed 
akonadi_kolab_resource_0 (0x7fef5c0148b0) 3764 BEGIN 
akonadi_kolab_resource_0 (0x7fef5c0148b0) 3764 OK Begin completed 
akonadi_kolab_resource_0 (0x7fef5c0148b0) 3765 UID STORE 3065 NOREV (REMOTEID "2" DIRTY false) 
akonadi_kolab_resource_0 (0x7fef5c0148b0) * 3065 FETCH (REV 0) 
akonadi_kolab_resource_0 (0x7fef5c0148b0) 3765 OK DATETIME "19-Apr-2016 10:06:26 +0000" STORE completed 
akonadi_kolab_resource_0 (0x7fef5c0148b0) 3767 COMMIT 
akonadi_kolab_resource_0 (0x7fef5c0148b0) 3767 OK Commit completed 
AkonadiConsole Browser Widget (0x7feef8027530) 27 UID FETCH 3065 FULLPAYLOAD CACHEONLY EXTERNALPAYLOAD (UID COLLECTIONID FLAGS SIZE REMOTEID REMOTEREVISION DATETIME) 
AkonadiConsole Browser Widget (0x7feef8027530) * 3065 FETCH (UID 3065 REV 0 REMOTEID "2" MIMETYPE "message/rfc822" COLLECTIONID 51 SIZE 5545 DATETIME "19-Apr-2016 10:06:26 +0000" FLAGS ($ATTACHMENT \SEEN) PLD:ENVELOPE[1] {227} ("Mon, 18 Apr 2016 16:49:21 +0200" "Booooooooooooooooooooooooooooooo" (("Jane Doe" NIL "doe2" "example.org")) NIL NIL (("John Doe" NIL "john.doe" "example.org")) NIL NIL NIL "<20160418145015.59B5F63E43@kolab1.example.org>" NIL) PLD:HEAD[1] {1031} Return-Path: <doe2@example.org> Received: from kolab1.example.org ([unix socket]) by kolab1.example.org (Cyrus 2.5.2.3-Kolab-2.5.2-5.el7.kolab_3.5) with LMTPA; Mon, 18 Apr 2016 14:50:15 +0000 X-Sieve: CMU Sieve 2.4 X-Virus-Scanned: amavisd-new at example.org X-Spam-Flag: NO X-Spam-Score: 0.242 X-Spam-Level: X-Spam-Status: No, score=0.242 tagged_above=-10 required=6.2 tests=[ALL_TRUSTED=-1, MISSING_MID=0.14, TRACKER_ID=1.102] autolearn=no autolearn_force=no Received: from 205d6d77f0aa.localnet (unknown [172.17.0.4]) by kolab1.example.org (Postfix) with ESMTPSA id 7D6EB63E42 for <john.doe@example.org>; Mon, 18 Apr 2016 14:49:31 +0000 (UTC) User-Agent: KOrganizer/4.13 (Linux/4.4.6-300.fc23.x86_64; KDE/4.14.3; x86_64; ) From: Jane Doe <doe2@example.org> To: John Doe <john.doe@example.org> Date: Mon, 18 Apr 2016 16:49:21 +0200 Subject: Booooooooooooooooooooooooooooooo Content-Type: multipart/mixed; boundary="nextPart4337192.OvIrPZzYpk" MIME-Version: 1.0 Message-Id: <20160418145015.59B5F63E43@kolab1.example.org> PLD:RFC822[1] [FILE] {7} 3210_r0) 
AkonadiConsole Browser Widget (0x7feef8027530) 27 OK UID FETCH completed 
akonadiconsole-489035169 (0x7fef240d0130) 32 UID FETCH 3065 FULLPAYLOAD EXTERNALPAYLOAD (UID COLLECTIONID FLAGS SIZE REMOTEID REMOTEREVISION DATETIME ATR:ErrorAttribute ATR:ScamAttribute ATR:MessageDisplayFormatAttribute) 
AkonadiConsole Browser Widget (0x7feef8027530) 28 UID FETCH 3065 FULLPAYLOAD CACHEONLY EXTERNALPAYLOAD (UID COLLECTIONID FLAGS SIZE REMOTEID REMOTEREVISION DATETIME) 
AkonadiConsole Browser Widget (0x7feef8027530) * 3065 FETCH (UID 3065 REV 0 REMOTEID "2" MIMETYPE "message/rfc822" COLLECTIONID 51 SIZE 5545 DATETIME "19-Apr-2016 10:06:26 +0000" FLAGS ($ATTACHMENT \SEEN) PLD:ENVELOPE[1] {227} ("Mon, 18 Apr 2016 16:49:21 +0200" "Booooooooooooooooooooooooooooooo" (("Jane Doe" NIL "doe2" "example.org")) NIL NIL (("John Doe" NIL "john.doe" "example.org")) NIL NIL NIL "<20160418145015.59B5F63E43@kolab1.example.org>" NIL) PLD:HEAD[1] {1031} Return-Path: <doe2@example.org> Received: from kolab1.example.org ([unix socket]) by kolab1.example.org (Cyrus 2.5.2.3-Kolab-2.5.2-5.el7.kolab_3.5) with LMTPA; Mon, 18 Apr 2016 14:50:15 +0000 X-Sieve: CMU Sieve 2.4 X-Virus-Scanned: amavisd-new at example.org X-Spam-Flag: NO X-Spam-Score: 0.242 X-Spam-Level: X-Spam-Status: No, score=0.242 tagged_above=-10 required=6.2 tests=[ALL_TRUSTED=-1, MISSING_MID=0.14, TRACKER_ID=1.102] autolearn=no autolearn_force=no Received: from 205d6d77f0aa.localnet (unknown [172.17.0.4]) by kolab1.example.org (Postfix) with ESMTPSA id 7D6EB63E42 for <john.doe@example.org>; Mon, 18 Apr 2016 14:49:31 +0000 (UTC) User-Agent: KOrganizer/4.13 (Linux/4.4.6-300.fc23.x86_64; KDE/4.14.3; x86_64; ) From: Jane Doe <doe2@example.org> To: John Doe <john.doe@example.org> Date: Mon, 18 Apr 2016 16:49:21 +0200 Subject: Booooooooooooooooooooooooooooooo Content-Type: multipart/mixed; boundary="nextPart4337192.OvIrPZzYpk" MIME-Version: 1.0 Message-Id: <20160418145015.59B5F63E43@kolab1.example.org> PLD:RFC822[1] [FILE] {7} 3210_r0) 
akonadiconsole-489035169 (0x7fef240d0130) * 3065 FETCH (UID 3065 REV 0 REMOTEID "2" MIMETYPE "message/rfc822" COLLECTIONID 51 SIZE 5545 DATETIME "19-Apr-2016 10:06:26 +0000" FLAGS ($ATTACHMENT \SEEN) PLD:HEAD[1] {1031} Return-Path: <doe2@example.org> Received: from kolab1.example.org ([unix socket]) by kolab1.example.org (Cyrus 2.5.2.3-Kolab-2.5.2-5.el7.kolab_3.5) with LMTPA; Mon, 18 Apr 2016 14:50:15 +0000 X-Sieve: CMU Sieve 2.4 X-Virus-Scanned: amavisd-new at example.org X-Spam-Flag: NO X-Spam-Score: 0.242 X-Spam-Level: X-Spam-Status: No, score=0.242 tagged_above=-10 required=6.2 tests=[ALL_TRUSTED=-1, MISSING_MID=0.14, TRACKER_ID=1.102] autolearn=no autolearn_force=no Received: from 205d6d77f0aa.localnet (unknown [172.17.0.4]) by kolab1.example.org (Postfix) with ESMTPSA id 7D6EB63E42 for <john.doe@example.org>; Mon, 18 Apr 2016 14:49:31 +0000 (UTC) User-Agent: KOrganizer/4.13 (Linux/4.4.6-300.fc23.x86_64; KDE/4.14.3; x86_64; ) From: Jane Doe <doe2@example.org> To: John Doe <john.doe@example.org> Date: Mon, 18 Apr 2016 16:49:21 +0200 Subject: Booooooooooooooooooooooooooooooo Content-Type: multipart/mixed; boundary="nextPart4337192.OvIrPZzYpk" MIME-Version: 1.0 Message-Id: <20160418145015.59B5F63E43@kolab1.example.org> PLD:ENVELOPE[1] {227} ("Mon, 18 Apr 2016 16:49:21 +0200" "Booooooooooooooooooooooooooooooo" (("Jane Doe" NIL "doe2" "example.org")) NIL NIL (("John Doe" NIL "john.doe" "example.org")) NIL NIL NIL "<20160418145015.59B5F63E43@kolab1.example.org>" NIL) PLD:RFC822[1] [FILE] {7} 3210_r0) 
AkonadiConsole Browser Widget (0x7feef8027530) 28 OK UID FETCH completed 
akonadiconsole-489035169 (0x7fef240d0130) 32 OK UID FETCH completed 
akonadiconsole-489035169 (0x7fef240d0130) 33 UID FETCH 3065 FULLPAYLOAD ALLATTR EXTERNALPAYLOAD (UID COLLECTIONID FLAGS SIZE REMOTEID REMOTEREVISION DATETIME) 
akonadiconsole-489035169 (0x7fef240d0130) * 3065 FETCH (UID 3065 REV 0 REMOTEID "2" MIMETYPE "message/rfc822" COLLECTIONID 51 SIZE 5545 DATETIME "19-Apr-2016 10:06:26 +0000" FLAGS ($ATTACHMENT \SEEN) PLD:HEAD[1] {1031} Return-Path: <doe2@example.org> Received: from kolab1.example.org ([unix socket]) by kolab1.example.org (Cyrus 2.5.2.3-Kolab-2.5.2-5.el7.kolab_3.5) with LMTPA; Mon, 18 Apr 2016 14:50:15 +0000 X-Sieve: CMU Sieve 2.4 X-Virus-Scanned: amavisd-new at example.org X-Spam-Flag: NO X-Spam-Score: 0.242 X-Spam-Level: X-Spam-Status: No, score=0.242 tagged_above=-10 required=6.2 tests=[ALL_TRUSTED=-1, MISSING_MID=0.14, TRACKER_ID=1.102] autolearn=no autolearn_force=no Received: from 205d6d77f0aa.localnet (unknown [172.17.0.4]) by kolab1.example.org (Postfix) with ESMTPSA id 7D6EB63E42 for <john.doe@example.org>; Mon, 18 Apr 2016 14:49:31 +0000 (UTC) User-Agent: KOrganizer/4.13 (Linux/4.4.6-300.fc23.x86_64; KDE/4.14.3; x86_64; ) From: Jane Doe <doe2@example.org> To: John Doe <john.doe@example.org> Date: Mon, 18 Apr 2016 16:49:21 +0200 Subject: Booooooooooooooooooooooooooooooo Content-Type: multipart/mixed; boundary="nextPart4337192.OvIrPZzYpk" MIME-Version: 1.0 Message-Id: <20160418145015.59B5F63E43@kolab1.example.org> PLD:ENVELOPE[1] {227} ("Mon, 18 Apr 2016 16:49:21 +0200" "Booooooooooooooooooooooooooooooo" (("Jane Doe" NIL "doe2" "example.org")) NIL NIL (("John Doe" NIL "john.doe" "example.org")) NIL NIL NIL "<20160418145015.59B5F63E43@kolab1.example.org>" NIL) PLD:RFC822[1] [FILE] {7} 3210_r0 ATR:MDNStateAttribute {1} I) 
akonadiconsole-489035169 (0x7fef240d0130) 33 OK UID FETCH completed 
MailFilter Kernel ETM (0x7fef3c014c30) 360 STATUS 51 (MESSAGES UNSEEN SIZE) 
Archive Mail Kernel ETM (0x7fef30014630) 360 STATUS 51 (MESSAGES UNSEEN SIZE) 
Archive Mail Kernel ETM (0x7fef30014630) * STATUS "Kolab/Archive/sub" (MESSAGES 2 UNSEEN 0 SIZE 11090) 
Archive Mail Kernel ETM (0x7fef30014630) 360 OK STATUS completed 
MailFilter Kernel ETM (0x7fef3c014c30) * STATUS "Kolab/Archive/sub" (MESSAGES 2 UNSEEN 0 SIZE 11090) 
MailFilter Kernel ETM (0x7fef3c014c30) 360 OK STATUS completed 
Archive Mail Kernel ETM (0x7fef30014630) 361 STATUS 28 (MESSAGES UNSEEN SIZE) 
MailFilter Kernel ETM (0x7fef3c014c30) 361 STATUS 28 (MESSAGES UNSEEN SIZE) 
Archive Mail Kernel ETM (0x7fef30014630) * STATUS "Kolab/Archive" (MESSAGES 0 UNSEEN 0 SIZE 0) 
Archive Mail Kernel ETM (0x7fef30014630) 361 OK STATUS completed 
MailFilter Kernel ETM (0x7fef3c014c30) * STATUS "Kolab/Archive" (MESSAGES 0 UNSEEN 0 SIZE 0) 
MailFilter Kernel ETM (0x7fef3c014c30) 361 OK STATUS completed 
AkonadiConsole Browser Widget (0x7feef8027530) 29 STATUS 51 (MESSAGES UNSEEN SIZE) 
AkonadiConsole Browser Widget (0x7feef8027530) * STATUS "Kolab/Archive/sub" (MESSAGES 2 UNSEEN 0 SIZE 11090) 
AkonadiConsole Browser Widget (0x7feef8027530) 29 OK STATUS completed 
AkonadiConsole Browser Widget (0x7feef8027530) 30 STATUS 28 (MESSAGES UNSEEN SIZE) 
AkonadiConsole Browser Widget (0x7feef8027530) * STATUS "Kolab/Archive" (MESSAGES 0 UNSEEN 0 SIZE 0) 
AkonadiConsole Browser Widget (0x7feef8027530) 30 OK STATUS completed

NotificationManager::notify ( Items ((3065,1,message/rfc822)) from collection 28 moved to collection 51 )
NotificationManager::notify ( Items ((3065,2,message/rfc822)) in collection 51 modified parts (REMOTEID) )

Because both folders only had one message inside, both items get the same REMOTEID for some time, until the imap resource has replayed the change and sets the REMOTEID to 2. If that fails however we end up with two messages with the same REMOTEID.

"akonadictl fsck" can now resolve duplicate remoteid's. In the worst case this will duplicate the messages on the server (to ensure no data is lost).
MOVE now clears the remoteId. This means that if the changereplay fails for some reason we'll end up with a message with an emtpy remote id (and akonadictl fsck can clean that up).

It would be relatively easy to prevent this issue altogether:

diff --git a/server/src/storage/akonadidb.xml b/server/src/storage/akonadidb.xml
index 5078a95..2ffceb5 100644
--- a/server/src/storage/akonadidb.xml
+++ b/server/src/storage/akonadidb.xml
@@ -66,7 +66,7 @@
   <table name="SchemaVersion">
     <comment>Contains the schema version of the database.</comment>
     <column name="version" type="int" default="0" allowNull="false"/>
-    <data columns="version" values="29"/>
+    <data columns="version" values="30"/>
   </table>

   <table name="Resource">
@@ -136,6 +136,7 @@
     <index name="collectionIndex" columns="collectionId" unique="false"/>
     <index name="gidIndex" columns="gid" unique="false"/>
     <index name="ridIndex" columns="remoteId" unique="false"/>
+    <index name="ridAndCollectionIndex" columns="remoteId,collectionId" unique="true"/>
     <reference name="parts" table="Part" key="pimItemId"/>
   </table>

but for this we'd require a cleanup of the database first.

The branch "dev/enforeUniqueRID" contains an implementation of the migration and constraint on the db level.

We can't implement the constraint as currently implemented because the constraint is violated in search collections (marked with isVirtual). These collections pull items together from multiple resources and can thus have multiple items with the same RID.

If we add both storageCollection and parentCollection to the index it should remain unique.

Nevermind the issue with search collections. The item is only linked into the collection, pimitemtable doesn't actually contain anything different, so the original idea should work just fine.

I'm running this patch from Kontact:4.13;Git, and it seems to work just fine. If no issues arise this will be rolled out to Kontact:4.13:Development

mollekopf closed this task as Resolved.Jun 28 2016, 9:56 AM

mollekopf claimed this task.

Avoid duplicates by GID/RIDClosed, ResolvedPublicActions

Description

Details

Related Objects

Event Timeline

Avoid duplicates by GID/RID
Closed, ResolvedPublic
Actions