Page MenuHomePhorge

IMAP and DAV migration
AbandonedPublic

Authored by mollekopf on Jul 1 2024, 2:57 PM.
Tags
None
Referenced Files
Unknown Object (File)
Mon, Oct 28, 4:05 PM
Unknown Object (File)
Sun, Oct 27, 2:35 PM
Unknown Object (File)
Thu, Oct 17, 1:10 AM
Unknown Object (File)
Sep 19 2024, 3:57 AM
Unknown Object (File)
Sep 19 2024, 3:01 AM
Unknown Object (File)
Sep 16 2024, 9:26 PM
Unknown Object (File)
Sep 15 2024, 12:57 AM
Unknown Object (File)
Sep 14 2024, 11:29 PM
Subscribers

Details

Reviewers
None
Group Reviewers
Restricted Project

Diff Detail

Repository
rK kolab
Branch
dev/mollekopf
Lint
Lint Skipped
Unit
No Test Coverage
Build Status
Buildable 47970
Build 18192: arc lint + arc unit

Event Timeline

mollekopf created this revision.

This only implements migrating imap messages but highlights how the current framework doesn't cleanly facilitate not using the job per item strategy.
I think there's a cleaner approach to implementing the two strategies than what I did above, e.g. by moving the processFolder logic entirely to the exporter.

mollekopf added a reviewer: Restricted Project.Jul 1 2024, 3:23 PM
machniak added inline comments.
src/app/DataMigrator/Kolab.php
180 ↗(On Diff #13818)

I think we should probably do this in steps, e.g. 50 or 100 messages per job. E.g. get all UIDs in folder and invoke a job for every 50 of them.

192 ↗(On Diff #13818)

A corner case, but Message-Id header may not exist. Maybe fallback to md5(Date.From.To.Subject). Also, I guess we'd need to handle \Seen state change (and probably all other flags).

196 ↗(On Diff #13818)

There is handlePartBody() that can be used to get the full message source in a memory efficient way using streams.

src/include/rcube_imap_generic.php
29

This is not needed, but you probably have to run composer dump-autoload.

mollekopf retitled this revision from Shoehorn the kolab migration into the datamigrator to IMAP and DAV migration.

dav migration, separate migrators for imap/dav

Good progress, but I was about to work on DAV, and I already have some conflicting code. Please, don't work on DAV at this moment.

Small imap fixup, let the exporter implement the migration strategy

So EWS can do per item jobs, and IMAP not.

Good progress, but I was about to work on DAV, and I already have some conflicting code. Please, don't work on DAV at this moment.

I though we discussed on monday that I would finish with dav... Anyways, if you like you can just take this over. The basics seem to work for me, but on the dav side I only looked at events (for addressbooks/todos the serialization is missing).
From my side this is good enough for a merge though.

For production I think we'll have to introduce a secondary job processing queue for long-running jobs.

In D4845: IMAP data migrator and other improvements I took this differential of yours, but I re-implemented some things differently. I think it's cleaner. Still a lot of TODO lines there. So, we might improve while writing tests. I have also an idea to try to do mail streaming for best performance.

In favor of the other diff