Page MenuHomePhorge

Improve Support For Multiple Calendars Via ActiveSync
Closed, ResolvedPublic

Description

Android phones used to only support one calendar via ActiveSync. Therefore, we introduced a white-list for devices that support multiple calendars and for all others we merge all calendar's events into one calendar.

Nowadays, most devices (even recent Android phones) do support multiple calendars and users frequently run into problems with only seeing one calendar.

Therefore, we should improve our support for multiple calendars. I see these options:

  1. Switch from a white-list to a black-list
  2. Make Folder-Merging configurable in ActiveSync settings
  3. Scrape Folder-Merging altogether

Details

Ticket Type
Task

Event Timeline

grote raised the priority of this task from to 60.
grote updated the task description. (Show Details)
grote changed Ticket Type from Task to Task.
grote added subscribers: grote, machniak.

This is also relevant for Outlook clients connected by ActiveSync. Is there any news on this task?

So, I tested some clients with my blacklist implementation and enabled folders hierarchies for all.

  1. Android 6.0:
    • Mail app supports hierarchies, but it does not display folders with non-existing parent
    • Calendar app displays all folders as top-level folders, including these with non-existing parent,
    • in Contacts app I see no way to see/select folders.
  1. iPad (iOS 7.0.6):
    • displays folders in a hierarchized way only in Mail app. Notes, Calendar, Tasks, Contacts display subfolders as top-level folders.
    • folders with non-existing parent are displayed as top-level folders, except for Mail app which does not display them at all.
  1. iPad (Symantec Secure Email app):
    • same as 2. with one difference. It displays hierarchies in Contacts, Calendar, Notes and Tasks.
  1. Outlook 2013:
    • never shows folders with non-existing parent
    • displays hierarchies only in Mail and Folders sections, in other places the list of folders is flat.
    • yes, it supports multiple calendars

I updated the list in my previous comment with Outlook 2013 behavior. Now, we need to think what we can do to make the behavior better.

So, I tested some clients with my blacklist implementation and enabled folders hierarchies for all.
[...]

Thank you for that update.

Could you explain what "folders with non-existing parent" means? I assume it is not top-level folders, because you also mention those in some of the other points. So is it folders that had a parent, but then the parent got deleted?

Not only deleted, but indeed IMAP allows for hierarchies with deleted parents. It may also exist but be unsubscribed (in general or for activesync only), or it may be a namespace root (which is also a kind of special folder and generally not subscibeable).

Not only deleted, but indeed IMAP allows for hierarchies with deleted parents. It may also exist but be unsubscribed (in general or for activesync only), or it may be a namespace root (which is also a kind of special folder and generally not subscibeable).

Thank you for clarifying.

So if I understand correctly we can have situations like

Calendar
    ├── [Subcalendar]
    │   └── Subsubcalendar
    ├── Subcalendar2
    ├── (Subcalendar3) 
    │   └── Subsubcalendar2
    └── Subcalendar4
           └── Subsubcalendar3

where Subcalendar would be deleted and Subcalendar would be unsubscribed. All other folders would be subscribed and we would expect them on the device.

But we're only seeing Calendar, Subcalendar2, Subcalendar4 and Subsubcalendar3 because both Subsubcalendar and Subsubcalendar2 are being hidden by the state of their parent folder and this behaviour is consistent across all devices?

Calendar would be the top level folder in this case.

Correct so far?

What I do not understand right now is the namespace root question. Can you give an example of that, perhaps?

In T224#31227, @greve wrote:

But we're only seeing Calendar, Subcalendar2, Subcalendar4 and Subsubcalendar3 because both Subsubcalendar and Subsubcalendar2 are being hidden by the state of their parent folder and this behaviour is consistent across all devices?

Not consistent. According to my previous findings it would be:

  1. Android: Flat list (no hierarchy): All except Subcalendar and Subcalendar3
  2. iOS: the same as Android
  3. Symantec app: All folders as in iOS, but with hierarchy.
  4. Oultook: Flat list (no hierarchy): Calendar, Subcalendar2, Subcalendar4, Subsubcalendar3.

Note, that for mail folders it will look different.

What I do not understand right now is the namespace root question. Can you give an example of that, perhaps?

e.g. Other Users/jane.doe/Calendar or Shared Folders/shared/Calendar. So, in this case you can subscribe both Calendar folders, but not their parents. Also, note that on some devices both Calendar folders will be displayed with the same name, which is a problem in itself. Outlook will not display the at all.

In T224#31227, @greve wrote:

But we're only seeing Calendar, Subcalendar2, Subcalendar4 and Subsubcalendar3 because both Subsubcalendar and Subsubcalendar2 are being hidden by the state of their parent folder and this behaviour is consistent across all devices?

Not consistent. According to my previous findings it would be:

  1. Android: Flat list (no hierarchy): All except Subcalendar and Subcalendar3
  2. iOS: the same as Android
  3. Symantec app: All folders as in iOS, but with hierarchy.
  4. Oultook: Flat list (no hierarchy): Calendar, Subcalendar2, Subcalendar4, Subsubcalendar3.

Note, that for mail folders it will look different.

Alright, thanks for that clarification. Much clearer now.

What I do not understand right now is the namespace root question. Can you give an example of that, perhaps?

e.g. Other Users/jane.doe/Calendar or Shared Folders/shared/Calendar. So, in this case you can subscribe both Calendar folders, but not their parents. Also, note that on some devices both Calendar folders will be displayed with the same name, which is a problem in itself. Outlook will not display the at all.

That's what I suspected. Thank you for confirming and elaborating.

Problem Space

  • Case 1:
Calendar
    ├── [Subcalendar]
    │   └── Subsubcalendar
    ├── Subcalendar2
    ├── (Subcalendar3) 
    │   └── Subsubcalendar2
    └── Subcalendar4
           └── Subsubcalendar3

Where [Subcalendar] is deleted, and (Subcalendar3) is unsubscribed.

  • Case 2:
Other Users
  └── jane.doe
    └── Calendar

where Other Users is a different root name space with jane.doe as the user, both cannot be subscribed to.

  • Case 3:
Shared Folders
   └── shared
    └── Calendar

where Shared Folders is a different root name space with shared as the prefix, both of which cannot be subscribed.

Current Behaviour

OSCase 1Case 2Case 3
AndroidMissing Subcalendar2 & Subcalendar3, FLAT / NO HIERARCHYNONO
iOSFlat list, no hierarchyNONO
Symantec AppMissing Subcalendar2 & Subcalendar3NONO
Outlook 2013+Missing Subcalendar2 & Subcalendar3NONO

Is this about correct? Can you please fix the table, if needed?

OSCase 1Case 2 and Case 3
AndroidMissing Subcalendar & Subcalendar3, FLAT / NO HIERARCHYNO HIERARCHY, only Calendar
iOSMissing Subcalendar & Subcalendar3, FLAT / NO HIERARCHYNO HIERARCHY, only Calendar
Symantec AppMissing Subcalendar & Subcalendar3only Calendar
Outlook 2013+Missing Subcalendar, Subcalendar3, Subsubcalendar, Subsubcalendar2, FLAT / NO HIERARCHYNo folders at all

Thank you, @machniak.

Does the "only Calendar" mean it shows only the main folder, or that it only shows Calendars, incuding hierarchy for the Symantec App?

So here is my take on this:

  • Subcalendar & Subcalendar3 do not exist / are not subscribed: So if they are not shown, that seems correct as long as Subsubcalendar, Subsubcalendar2 are shown.
  • Outlook is the only client that fails in this case. Given this is the only client, the reason for failure seems to be client side.
  • Given that Symantic App is showing hierarchy, the lack of hierarchy also seems client side

So Case 1 seems about mitigating client side "failure" in the sense of "not helpful inconsistencies between clients."

Considering the loss of hierarchy in several clients, we probably also want to mitigate that for usability purposes.

From user story, I think the best user experience would likely be if we could set the full path (as it would be shown on server when folder listing) as folder name.

So in the example above,

FolderMaps to name on device
CalendarCalendar
SubsubcalendarCalendar/../Subsubcalendar
Subcalendar2Calendar/Subcalendar2
Subsubcalendar2Calendar/../Subsubcalendar2
Subcalendar4Calendar/Subcalendar4
Subsubcalendar3Calendar/Subcalendar4/Subsubcalendar3

Would such folder names be possible in the ActiveSync protocol? This would address the issue on Android, iOS and Outlook for Case 1, and should not totally destroy the usability for the Symantec App (one would hope).

In the worst case, this would become an option to turn this off for a device only for Case 1 (it would be on by default) and for Calendars for Case 2 and Case 3.

Otherwise Case 2 and Case 3 should prefix an additional "<username>/" for Case 2 and "shared/" for Case 3 and serve them as regular folders to the users own folders.

Please note this says nothing about technical feasibility, only the desired result from a usability perspective...

What do you think, @machniak?

In T224#31251, @greve wrote:

Does the "only Calendar" mean it shows only the main folder, or that it only shows Calendars, incuding hierarchy for the Symantec App?

It would display only the Calendar folder, without any hierarchy in this case.

Would such folder names be possible in the ActiveSync protocol?

We can set anything as a folder name.

In T224#31251, @greve wrote:

Does the "only Calendar" mean it shows only the main folder, or that it only shows Calendars, incuding hierarchy for the Symantec App?

It would display only the Calendar folder, without any hierarchy in this case.

Would it display them all, or just the first/primary/top level one?

Would such folder names be possible in the ActiveSync protocol?

We can set anything as a folder name.

Alright.

So that sounds like such "emulate hierarchy" folder naming feature could in fact work?

Given the inconsistencies between devices, this would probably require to be a "generate folder name" function that can do this

  • per device
  • per type
  • per name space

as required. It's not beautiful... but that seems to be a consequence of device inconsistencies.

Or is there a way to fix this on the ActiveSync level that would work for all devices?

In T224#31305, @greve wrote:

Would it display them all, or just the first/primary/top level one?

It would display only the Calendar folder as a top-level folder. But that's only one of many activesync clients, who knows what others do.

So that sounds like such "emulate hierarchy" folder naming feature could in fact work?

Given the inconsistencies between devices, this would probably require to be a "generate folder name" function that can do this

I'd call it "flat (or flattened) folders list". There will be one more issue to solve with this aproach. Folder renaming. I'd say we should not allow renaming these "flattened folders" over ActiveSync.

Or is there a way to fix this on the ActiveSync level that would work for all devices?

No. For devices that do not support hierarchies we can do nothing more. For those supporting it there could be another aproach. E.g. we could return these "missing" folders as they were subscribed, allowing devices to display the full hierarchy, but making those folders unaccessible (by returning an error on access). I'm not sure how devices would react to such errors. We also could consider doing the same for mail folders, even if e.g. calendars are in "flat" mode.

Considering all, for simplicity I'd say we should implement it this way:

  • for everything but mail - use "flat mode" for every folder
  • for mail - use "flat mode" for every folder that has missing parent

Which means that for such calendar folders hierarchy:

Calendar
    ├── [Subcalendar]
    │   └── Subsubcalendar
    ├── Subcalendar2
    ├── (Subcalendar3) 
    │   └── Subsubcalendar2
    └── Subcalendar4
           └── Subsubcalendar3
Other Users
  └── jane.doe
    └── Calendar
Shared Folders
   └── shared
    └── Calendar

we end up with:

Calendar
Calendar/../Subsubcalendar
Calendar/Subcalendar2
Calendar/../Subsubcalendar2
Calendar/Subcalendar4
Calendar/Subcalenda4/Subsubcalendar3
jane.doe/Calendar
shared/Calendar

For mail folders hierarchy:

INBOX
Mail
    ├── [Submail]
    │   └── Subsubmail
    ├── Submail2
    ├── (Submail3) 
    │   └── Subsubmail2
    └── Submail4
           └── Subsubmail3
Other Users
  └── jane.doe
    └── Mail
Shared Folders
   └── shared
    └── Mail

it would result in:

INBOX
Mail
    ├── ../Subsubmail
    ├── Submail2
    ├── ../Subsubmail2
    └── Submail4
           └── Subsubmail3
jane.doe/Mail
shared/Mail

Last thing is the way we "mark" the non-existing folders. Is ../ not confusing? Maybe better to use the folder name in [] brackets, e.g. Calendar/[Subcalendar]/Subsubcalendar or [jane.doe]/Mail. On the other hand too long names may be not displayed/truncated on the device or sth.

In T224#31305, @greve wrote:

Would it display them all, or just the first/primary/top level one?

It would display only the Calendar folder as a top-level folder.

That's what I feared.

But that's only one of many activesync clients, who knows what others do.

Exactly. Extremely annoying,

So that sounds like such "emulate hierarchy" folder naming feature could in fact work?

Given the inconsistencies between devices, this would probably require to be a "generate folder name" function that can do this

I'd call it "flat (or flattened) folders list". There will be one more issue to solve with this aproach. Folder renaming. I'd say we should not allow renaming these "flattened folders" over ActiveSync.

Sounds sensible.

But if there is a rename on the server side, we'd have to catch that and deal with it in a robust way even if that mean a "delete" and "new" for the mobile device.

Or is there a way to fix this on the ActiveSync level that would work for all devices?

No. For devices that do not support hierarchies we can do nothing more.

Alright, thank you for confirming.

For those supporting it there could be another aproach. E.g. we could return these "missing" folders as they were subscribed, allowing devices to display the full hierarchy, but making those folders unaccessible (by returning an error on access). I'm not sure how devices would react to such errors.

Would that not make things even more complex?

Considering all, for simplicity I'd say we should implement it this way:

  • for everything but mail - use "flat mode" for every folder

Would devices that support hierarchy then not show this as

Calendar

├── Calendar/Subcalendar
└── Calendar/Subcalendar2

and so on? I was thinking where hierarchy is working we probably want to preserve that.

  • for mail - use "flat mode" for every folder that has missing parent

Which means that for such calendar folders hierarchy:

That seems okay.

Last thing is the way we "mark" the non-existing folders. Is ../ not confusing? Maybe better to use the folder name in [] brackets, e.g. Calendar/[Subcalendar]/Subsubcalendar or [jane.doe]/Mail. On the other hand too long names may be not displayed/truncated on the device or sth.

Yeah, I was also concerned about length. And the folder is deactivated or deleted, so replacing it by ".." seemed okay as an approach.

But if people feel strongly about using the actual name in brackets, I can live with that, too...

greve added subscribers: seigo, vanmeeuwen.

It seems this should now get some thought and insight from @seigo and then go over to @vanmeeuwen to make the call on how to proceed. Assigning to @seigo first.

I'd say we should not allow renaming these "flattened folders" over ActiveSync.

That's the easy way, but also very annoying for anyone using ActiveSync as their primary interface to Kolab. That would be everyone using Outlook in future.

Parsing out paths is not hard to do, and calculating the necessary moves and/or folder creation is similarly not very difficult. The first is string manipulation (splitting a string into a list on non-escaped /s), the latter is applying that result to the folders in Kolab.

Last thing is the way we "mark" the non-existing folders. Is ../ not confusing? Maybe better to use the folder name in [] brackets

Ellission is not great, since if you have two calendars with the same name under two non-extant folders, they will end up with the same name: "../SameName".

I'm not sure it matters at all to even denote that a folder does not exist, however. "Calendars/DoesNotExist/CalendarName" is fine, and would at least match what they see in e.g. the web interface.

So I would recommend just including all parts of the path as-is, regardless of folder existence, for flat display.

It would be quite nice to be able to white-list in clients that we know do support hierarchies and skip flattening all together. But maybe that is a second step.

That said, this is out of scope for this ticket. This task should be closed imho and new tasks started for:

  • hierarchy flattening
  • option (settable via Roundcube Settings) to make ActiveSync strictly follow folder subscription (per-device?)

I agree that we're bloating this ticket with considerations that go past the white- vs. blacklist changeover. I'll create a sub-task and break the inheritance.

vanmeeuwen claimed this task.

I believe I have provided what I thing is a coherent and comprehensive description of the considerations made in this ticket, and elsewhere, to the new tickets.

Closing this one.