Page MenuHomePhorge
Files F120825570

kolab-2.3.rst
No OneTemporary
Actions

Authored By
Unknown
Size
15 KB
Referenced Files
None
Subscribers
None

kolab-2.3.rst
View Options

========================
Upgrading from Kolab 2.3
========================
As Kolab Groupware used to be distributed as an OpenPKG stack, with different locations for files, the upgrade of a Kolab 2 server to a Kolab 3 server is a largely manual process.
.. NOTE::
The Kolab 3 server used here has been set up for the same domain name space as the Kolab 2 server. No users have been created on the Kolab 3 server. kolab2.example.org refers to the Kolab 2 server, kolab3.example.org refers to the Kolab 3 server.
.. rubric:: Preparations
1. During the migration, the Kolab 2 server cannot be allowed to;
* Receive new email.
* Let users post/submit new data.
The easiest way to prevent any of this happening is to shut down all Kolab Groupware related services on kolab2.example.org:
.. parsed-literal::
# /kolab/bin/openpkg rc stop all
2. Create a backup of the data from kolab2.example.org.
3. Shut down the Cyrus IMAP service on kolab3.example.org:
.. parsed-literal::
# service cyrus-imapd stop
4. Shut down the Kolab daemon on kolab3.example.org:
.. parsed-literal::
# service kolabd stop
5. Update the settings related to the recipient policy in /etc/kolab/kolab.conf. The following settings are important:
* The primary_mail setting in the [$domain] section.
* The policy MUST[2] either match the former convention used, if any, or not be enabled at all. See Example 1.1, “Example Migration of example.org” for an example and some more gotchas.
.. WARNING::
In case the recipient policy is not to be applied, consider updating the user_types as per the instructions in Section 5.1, “Editing user_types”.
The secondary_mail setting in the [$domain] section.
This part of the policy does not apply should the primary_mail setting already have been disabled.
.. rubric:: Example Migration of example.org
Our first example has had a running Kolab 2.3 on OpenPKG server, with a general email address convention of "surname"@example.org.
By default, a Kolab 3.0 Groupware server will apply a recipient policy for the mail attribute value of "givenname"."surname"@example.org. The recipient policy must therefore be adjusted.
In the [example.org] section in /etc/kolab/kolab.conf, the primary_mail setting must be adjusted to match the "surname"@example.org convention:
.. parsed-literal::
(...snip...)
[example.org]
primary_mail = %(surname)s@%(domain)s
(...snip...)
Now, users that are created will get a mail attribute value of "surname"@example.org assigned.
First adding user John Doe will give him a mail attribute value of doe@example.org, but should you have a Jane Doe as well, she would get doe2@example.org. It is therefore important to add users in order.
Migration of LDAP
-----------------
This section has not been authored yet.
Migration and Upgrade of Data
-----------------------------
Migrate the Data Through Copying
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Login to kolab3.example.org to execute the steps in this procedure.
1. Copy mailboxes.db and annotations.db. These files are located in /kolab/var/imapd/ on your Kolab 2 server.
.. parsed-literal::
# scp root@kolab2.example.org:/kolab/var/imapd/annotations.db \\
/var/lib/imap/annotations.db
(...)
# scp root@kolab2.example.org:/kolab/var/imapd/mailboxes.db \\
/var/lib/imap/mailboxes.db
(...)
If cyrus can't start with these files with a message like the following, you need to convert the files instead of copying.
.. parsed-literal::
skiplist: invalid magic header: /var/lib/imap/mailboxes.db
(...)
DBERROR: opening /var/lib/imap/mailboxes.db: cyrusdb error
(...)
Use the following commands to convert the files from Berkeley DB to skiplist:
.. parsed-literal::
ssh root@kolab2.example.org "/kolab/bin/cvt_cyrusdb /kolab/var/imapd/annotations.db /tmp/annotations.db skiplist"
ssh root@kolab2.example.org "/kolab/bin/cvt_cyrusdb /kolab/var/imapd/mailboxes.db /tmp/mailboxes.db skiplist"
After the conversion, copy the files to the new kolab3 server:
.. parsed-literal::
# scp root@kolab2.example.org:/tmp/annotations.db /var/lib/imap/annotations.db
(...)
# scp root@kolab2.example.org:/tmp/mailboxes.db /var/lib/imap/mailboxes.db
(...)
2. Migrate the mail spool:
.. parsed-literal::
# rsync -rltpHvz --progress --partial \\
kolab2.example.org:/kolab/var/imapd/spool/ \\
/var/spool/imap/
(...)
3. Migrate the seen and subscription databases:
.. parsed-literal::
# rsync -rltpHvz --progress --partial \\
kolab2.example.org:/kolab/var/imapd/domain/ \\
/var/lib/imap/domain/
(...)
# rsync -rltpHvz --progress --partial \\
kolab2.example.org:/kolab/var/imapd/user/ \\
/var/lib/imap/user/
(...)
4. Ensure the filesystem permissions are correct:
.. parsed-literal::
# chown -R cyrus:mail /var/lib/imap/ /var/spool/imap/
5. Ensure only the cyrus user can read and write, and the mail group can read the contents of either directory tree:
.. parsed-literal::
# find /var/lib/imap -type f -exec chmod 640 {} \\;
# find /var/lib/imap -type d -exec chmod 750 {} \\;
# find /var/spool/imap -type f -print0 | xargs -0 chmod 640
# find /var/spool/imap -type d -print0 | xargs -0 chmod 750
If your find doesn't have the -print0 option or your xargs command doesn't know -0, you can use the much slower commands below:
.. parsed-literal::
# find /var/spool/imap -type f -exec chmod 640 {} \\;
# find /var/spool/imap -type d -exec chmod 750 {} \\;
6. As the seen databases are particularly hard for Cyrus IMAP 2.4 to upgrade real-time, convert the seen databases with, for example:
.. parsed-literal::
#!/bin/bash
find /var/lib/imap/ -type f -name "\*.seen" | \
sort | while read seendb; do
/usr/lib/cyrus-imapd/cvt_cyrusdb ${seendb} skiplist ${seendb}.txt flat
mv ${seendb} ${seendb}.orig
/usr/lib/cyrus-imapd/cvt_cyrusdb ${seendb}.txt flat ${seendb} skiplist
chown cyrus:mail ${seendb} ${seendb}.txt ${seendb}.orig
done
7. Stop the Kolab daemon:
.. parsed-literal::
# service kolabd stop
8. Start the Cyrus IMAP service on kolab3.example.org:
.. parsed-literal::
# service cyrus-imapd start
.. IMPORTANT::
Users should not yet be allowed to interact with the system at this point. We suggest closing access to the system through the firewall.
9. Prepare some environment variables
We need the cyrus-admin user and associated password for the following steps, so we put them in environment variables:
.. parsed-literal::
cyrus_admin=$(awk '/^admin_login/ { print $3 }' /etc/kolab/kolab.conf)
cyrus_admin_pw=$(awk '/^admin_password/ { print $3 }' /etc/kolab/kolab.conf)
passwd="$cyrus_admin_pw"
imap_host="localhost"
10. Upgrade the Format
Use the Cyrus IMAP administrator account to select all mailboxes, to make sure the format upgrade is not taking place while the user is attempting to login / select a mailbox.
Average sized mailboxes (those restricted with a reasonable quota such as 2GB) can take about 2 minutes to upgrade. A single folder can upgrade as fast as 5 seconds. Mailboxes that contain a lot of messages (such as a shared lists folder that also functions as an archive) can take up to 2-4 minutes to upgrade (approximately 40.000 messages).
.. NOTE::
Optionally, if errors are expected enable full so-called telemetry logging so that issues can be backtracked. On the IMAP server, issue the following commands:
.. parsed-literal::
# mkdir -p /var/lib/imap/log/cyrus-admin
# chown cyrus:mail /var/lib/imap/log/cyrus-admin
#!/bin/bash
kolab lm | while read folder; do
echo ". SELECT \\"${folder}\\""
done | \\
imtest -t "" \\
-u ${cyrus_admin} \\
-a ${cyrus_admin} \\
-w '${cyrus_admin_pw}' \\
${imap_host}
or, alternatively;
.. parsed-literal::
#!/bin/bash
. ./settings.sh
echo '. LIST "" "\*"' | \\
imtest \\
-t "" \\
-u ${cyrus_admin} \\
-a ${cyrus_admin} \\
-w '${cyrus_admin_pw}' \\
${imap_host} | \\
sed -r \\
-e '/^\\* LIST/!d' \\
-e 's/.\*\"\/\"\s(.\*)/\1/g' \\
-e 's/^"//g' \\
-e 's/"$//g' \\
-e 's/\s\*\r\*\n\*$//g' | \\
while read folder; do
echo ". SELECT \\"${folder}\\""
done | \\
imtest -t "" \\
-u ${cyrus_admin} \\
-a ${cyrus_admin} \\
-w '${cyrus_admin_pw}' \\
${imap_host}
On your console display, you will see some errors stating NO Permission Denied. These errors you can ignore.
You'll see messages such as the following appear in ``/var/log/maillog``:
.. parsed-literal::
Aug 8 16:40:10 kolab imap[4644]: Index upgrade: example.org!shared.lists.example^org.memo (10 -> 12)
Aug 8 16:40:10 kolab imap[4644]: seen_db: user cyrus-admin opened /var/lib/imap/user/c/cyrus-admin.seen
11. The annotations database may not have been upgraded correctly, causing some annotations to miss the first 4 characters of their value. The easiest way to fix the issue, that is known to work, is to get the annotation values as they were on the old (Kolab 2) IMAP server, and set them on the new (Kolab 3) IMAP server.
.. parsed-literal::
# kolab -c conf/kolab-kolab2.example.org.conf \\
list-mailbox-metadata "user/john.doe/Calendar\*@example.org"
Folder user/john.doe/Calendar@example.org
/shared/vendor/cmu/cyrus-imapd/lastpop
/shared/vendor/cmu/cyrus-imapd/partition default
/shared/vendor/cmu/cyrus-imapd/lastupdate 8-Aug-2012 16:16:06 +0200
/shared/vendor/cmu/cyrus-imapd/size 266564
/shared/vendor/cmu/cyrus-imapd/pop3newuidl true
/shared/vendor/cmu/cyrus-imapd/sharedseen false
/shared/vendor/kolab/folder-type event.default
/shared/vendor/cmu/cyrus-imapd/condstore true
/shared/vendor/cmu/cyrus-imapd/duplicatedeliver false
/shared/vendor/kolab/incidences-for admins
/shared/vendor/kolab/folder-test true
Folder user/john.doe/Calendar/Private@example.org
/shared/vendor/cmu/cyrus-imapd/lastpop
/shared/vendor/cmu/cyrus-imapd/partition default
/shared/vendor/cmu/cyrus-imapd/lastupdate 8-Aug-2012 16:08:58 +0200
/shared/vendor/cmu/cyrus-imapd/condstore true
/shared/vendor/cmu/cyrus-imapd/pop3newuidl true
/shared/vendor/cmu/cyrus-imapd/size 305426
/shared/vendor/cmu/cyrus-imapd/sharedseen false
/shared/vendor/kolab/folder-type event
/shared/vendor/cmu/cyrus-imapd/duplicatedeliver false
# kolab list-mailbox-metadata "user/john.doe/Calendar\*@example.org"
Folder user/john.doe/Calendar@example.org
/shared/vendor/cmu/cyrus-imapd/lastpop
/shared/vendor/cmu/cyrus-imapd/partition default
/shared/vendor/cmu/cyrus-imapd/lastupdate 8-Aug-2012 16:16:27 +0200
/shared/vendor/cmu/cyrus-imapd/duplicatedeliver false
/shared/vendor/cmu/cyrus-imapd/pop3newuidl true
/shared/vendor/cmu/cyrus-imapd/size 266564
/shared/vendor/cmu/cyrus-imapd/sharedseen false
/shared/vendor/kolab/folder-type event.default
Folder user/john.doe/Calendar/Private@example.org
/shared/vendor/cmu/cyrus-imapd/lastpop
/shared/vendor/cmu/cyrus-imapd/partition default
/shared/vendor/cmu/cyrus-imapd/lastupdate 8-Aug-2012 16:27:44 +0200
/shared/vendor/cmu/cyrus-imapd/duplicatedeliver false
/shared/vendor/cmu/cyrus-imapd/pop3newuidl true
/shared/vendor/cmu/cyrus-imapd/size 305426
/shared/vendor/cmu/cyrus-imapd/sharedseen false
Fix'em:
.. parsed-literal::
#!/bin/bash
# Interesting Annotations
declare -a ia
ia[${#ia[@]}]="/shared/vendor/kolab/folder-type"
ia[${#ia[@]}]="/shared/vendor/kolab/folder-test"
for folder_search in user/\*@example.org shared/\*@example.org; do
kolab -c conf/kolab-kolab.kolabsys.com.conf \\
list-mailbox-metadata "${folder_search}" | \\
while read line; do
if [ ! -z "$(echo $line | grep ^Folder)" ]; then
current_folder=$(echo $line | cut -d' ' -f2-)
echo "Folder: '${current_folder}'"
else
annotation_key=$(echo $line | awk '{print $1}')
annotation_value=$(echo $line | awk '{print $2}')
i=0
set_annotation=0
while [ $i -lt ${#ia[@]} ]; do
if [ "${ia[$i]}" == "${annotation_key}" ]; then
set_annotation=1
break
fi
let i++
done
if [ ${set_annotation} -eq 0 ]; then
continue
fi
echo "Setting ${annotation_key} to ${annotation_value}"
kolab set-mailbox-metadata \\
"${current_folder}" \\
"${annotation_key}" \\
"${annotation_value}"
fi
done
done
12. Upgrade all messages from Kolab Format version 2 to Kolab Format version 3 using kolab-formatupgrade. This command is run in two parts. The first will upgrade all mailbox contents in the personal namespace:
.. parsed-literal::
# kolab lm "user/%@example.org" | \\
sed -e 's/user\\///g' | \\
while read user; do
kolab-formatupgrade \\
--user "${user}" \\
--password $password \\
--proxyauth cyrus-admin \\
--port 143 \\
--encrypt TLS \\
localhost
done
12. The second part upgrades the contents of shared folders. Shared folders have no designated owners, and we can therefore not login as a designated user to upgrade the format.
As the user cyrus-admin normally does not have the necessary privileges to insert new messages into mail folders, so we're going to have to give out the rights first. We'll delete them again afterwards.
.. parsed-literal::
# kolab sam shared/\*@example.org cyrus-admin lrswiptexa
# kolab lm shared/\*@example.org | \\
while read folder; do
kolab-formatupgrade \\
--user cyrus-admin \\
--password $password \\
--port 143 \\
--encrypt TLS \\
--folder "${folder}" \\
localhost
done
# kolab dam shared/\*@example.org cyrus-admin

File Metadata

Mime Type
text/plain
Expires
Fri, Apr 24, 10:40 AM (1 d, 17 h ago)
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
18806715
Default Alt Text
kolab-2.3.rst (15 KB)

Event Timeline