Page MenuHomePhorge
Files F120825883

No OneTemporary
Actions

Authored By
Unknown
Size
38 KB
Referenced Files
None
Subscribers
None

View Options

diff --git a/doc/README.cyrusdb.md b/doc/README.cyrusdb.md
index 646a7ddd8..d2d630324 100644
--- a/doc/README.cyrusdb.md
+++ b/doc/README.cyrusdb.md
@@ -1,290 +1,290 @@
# The CyrusDB Interface
> This document is from an [email](https://lists.andrew.cmu.edu/pipermail/cyrus-devel/2018-February/004217.html) Bron sent to the [cyrus-devel](https://lists.andrew.cmu.edu/pipermail/cyrus-devel/) mailing list.
The cyrusdb interface is a common API for accessing key-value
datastores from within Cyrus code. It's part of the libcyrus shared
object (but not libcyrus_min), used by all Cyrus binaries. The use of
the cyrusdb API abstracts the details of the underlying data store,
allowing sites to use different database types depending on their
needs, and allowing Cyrus developers to use a common API for data
storage needs.
The entire cyrusdb source lives in the `lib/` directory of the
`cyrus-imapd` repository, in the following files:
- [`lib/cyrusdb.h`](https://github.com/cyrusimap/cyrus-imapd/blob/master/lib/cyrusdb.h)
- [`lib/cyrusdb.c`](https://github.com/cyrusimap/cyrus-imapd/blob/master/lib/cyrusdb.c)
- [`lib/cyrusdb_flat.c`](https://github.com/cyrusimap/cyrus-imapd/blob/master/lib/cyrusdb_flat.c)
- [`lib/cyrusdb_quotalegacy.c`](https://github.com/cyrusimap/cyrus-imapd/blob/master/lib/cyrusdb_quotalegacy.c)
- [`lib/cyrusdb_skiplist.c`](https://github.com/cyrusimap/cyrus-imapd/blob/master/lib/cyrusdb_skiplist.c)
- [`lib/cyrusdb_sql.c`](https://github.com/cyrusimap/cyrus-imapd/blob/master/lib/cyrusdb_sql.c)
- [`lib/cyrusdb_twoskip.c`](https://github.com/cyrusimap/cyrus-imapd/blob/master/lib/cyrusdb_twoskip.c)
### `cyrusdb.h`
* interface definitions (all access to cyrusdb databases happens through the functions defined here)
* the [`struct cyrusdb_backend`](https://github.com/cyrusimap/cyrus-imapd/blob/master/lib/cyrusdb.h#L94) data structure which defines the interface implemented by each backend.
* constants for flags to the [`cyrusdb_open`](https://github.com/cyrusimap/cyrus-imapd/blob/master/lib/cyrusdb.h#L251) call, and return codes. Cyrusdb functions all return their own `CYRUSDB_*` error codes, which are not compatible with the `r = IMAP_*` return codes used throughout much of the rest of the codebase.
### `cyrusdb.c`
* implementations of the wrapper functions around the backends, including default implementations of some functions which are common to many backends but overridden by some.
* a wrapper to initialise and cleanup the state of each backend (if needed) during Cyrus set up / tear down.
### `cyrusdb_*.c`
* the actual implementations of each backend! We'll look at some in a sec.
## Tools
There are also some tools to work with and support Cyrus databases:
### `imap/ctl_cyrusdb`
Performs maintenance on the cyrusdb subsystem. This is called in two places:
- START: `ctl_cyrusdb -r` (recovery). This is the *ONLY PLACE* that code is guaranteed to be run at startup on every Cyrus installation, so you'll find quite a lot of detritus has built up in this codepath over the years.
- - EVENTS: `ctl_cyrusdb -c` (checkpoint). This is run regularly (`period=180` at FastMail, examples in the codebase have `period=5` or `period=30`). Both this codepath and `cyr_expire` tend to run periodically on Cyrus systems, and cleanup code is spread between those two locations.
+ - EVENTS: `ctl_cyrusdb -c` (checkpoint). This is run regularly (`period=180` at Fastmail, examples in the codebase have `period=5` or `period=30`). Both this codepath and `cyr_expire` tend to run periodically on Cyrus systems, and cleanup code is spread between those two locations.
### `imap/cvt_cyrusdb`
Used for converting a database between versions. This is often used
to prepare for upgrade, particularly in the past when Cyrus supported
Berkeley DB which didn't upgrade cleanly across OS versions, it was
common to use `cvt_cyrusdb` to turn databases into a very portable
format (`flat` or `skiplist`) before upgrading, upgrade the OS,
convert back to the fast format (Berkeley DB) and then restart.
### `imap/cyr_dbtool`
Once known as `brontool`, this is the first piece of Cyrus code I ever wrote! It's a fairly dumb wrapper around the CyrusDB interface, and able to be used to read, write, or iterate any parts of a database. Its interactive mode is not special-character clean, but it can also be used in batch mode, which uses IMAP atom-string literal8 for input/output, and hence can roundtrip data reliably.
There are also tools like: `ctl_conversationsdb`, `dav_reconstruct`
and `ctl_mboxlist` which can be used to manage individual databases
through a more specific interface which understands the context as
well as just the raw key/value.
## How to use CyrusDB
Assuming that [`cyrus_init()`](https://github.com/cyrusimap/cyrus-imapd/blob/master/lib/cyrusdb.h#L247) has been called, which calls [`cyrusdb_init()`](https://github.com/cyrusimap/cyrus-imapd/blob/master/lib/cyrusdb.c#L328), you can assume that databases will work in any Cyrus code.
The first step is to open a database. Databases have a filename -
this might be a literal filename on the backend, a directly containing
data, or an opaque token used by the backend to locate a datasource.
```c
int flags = 0;
struct db *mydb = NULL;
int r = cyrusdb_open("skiplist", "/tmp/database.db", flags, &mydb);
if (!r) return mydb; // if (r == CYRUSDB_OK) { ... }
/* XXX: error handling */
```
#### Accepted flags
* `CYRUSDB_CREATE` - if the named database doesn't exist, create a
blank database.
* `CYRUSDB_MBOXSORT` - use the abomination called
`improved_mboxlist_sort` which re-orders a couple of characters to
allow "foo.bar" to sort before "foo bar", for perfectly good
reasons, but we're going to fix it a better way. Not every engine
supports arbitrary collation, and if many engines corrupt horribly
if the same database is opened with different choices for this flag.
Ouch.
* `CYRUSDB_CONVERT` - if set and the database fails to open, attempt a
magic detection on the file content and try to convert the database
to the requested backend type before opening it. In-place upgrades!
If this is NOT set, then instead the magic detection will still be
performed, but the open database will be returned using the correct
engine for that database rather than converted. Magic detection
only currently works for single-file database formats.
* `CYRUSDB_NOCOMPACT` - if the database format supports automatic
compacting, don't use it. Handy for when you want to read without
causing any possible issues (e.g. read-only filesystem during
recovery) or when performance is critical and you don't want to risk
waiting while a recompact happens.
All the remaining functions take that [`struct db`](https://github.com/cyrusimap/cyrus-imapd/blob/master/lib/cyrusdb.c#L83) pointer.
There's also
a
[`cyrusdb_lockopen()`](https://github.com/cyrusimap/cyrus-imapd/blob/master/lib/cyrusdb.c#L179) interface
which takes a transaction pointer and returns with the transaction
already active. This isn't actually being used yet, but is intended
to allow slightly more efficient single-operation database use. Right
now, open returns an unlocked database, but may need to lock as part
of the setup, so keeping that lock would avoid one extra unlock/lock
cycle.
## Reading, writing, transactions
CyrusDB supports both transactional and non-transactional access.
Transactions are always exclusive. This is arguably a deficiency in
the interface, particularly since many engines implement a
non-exclusive (read) lock internally anyway.
There are now 4 interfaces to read data. Two of which are original
cyrusdb and two of which are more recently added.
* original
- [`cyrusdb_fetch()`](https://github.com/cyrusimap/cyrus-imapd/blob/master/lib/cyrusdb.c#L194) - fetch a single value by exact key.
- [`cyrusdb_foreach()`](https://github.com/cyrusimap/cyrus-imapd/blob/master/lib/cyrusdb.c#L229) - given a prefix, iterate over all the keys with that prefix (including exactly that key) in order.
* newer:
- [`cyrusdb_fetchnext()`](https://github.com/cyrusimap/cyrus-imapd/blob/master/lib/cyrusdb.c#L216) - given an exact key, fetch the following key and value (regardless of whether the key exists), e.g given keys "f" and "g", fetchnext "foo" would return "g", as would fetchnext "f". This can be used to implement foreach (indeed, the skips do exactly that).
- [`cyrusdb_forone()`](https://github.com/cyrusimap/cyrus-imapd/blob/master/lib/cyrusdb.c#L241) - given an exact key, act like `cyrusdb_foreach` but only for that one key. This is a convenience wrapper around fetch to allow doing things like:
```c
r = cyrusdb_forone(mydb, "folder", 6, p, cb, rock, &tid);
if (!r) r = cyrusdb_foreach(mydb, "folder.", 7, p, cb, rock, &tid);
```
Which does precisely *"folder"* and its children without visiting any other keys that have *"folder"* as a prefix.
Since the cyrusdb interface always takes both a pointer and a length, it's also possible to use:
```c
char *key = "folder.";
r = cyrusdb_forone(mydb, key, 6, p, cb, rock, &tid);
if (!r) r = cyrusdb_foreach(mydb, key, 7, p, cb, rock, &tid);
```
You may have noticed that `tid` at the end. Every function for acting on the database takes as its last argument a [`struct txn **`](https://github.com/cyrusimap/cyrus-imapd/blob/master/lib/cyrusdb.h#L49). You can pass one of three things to this:
* `NULL` - non-transactional request. Do whatever you need for
internal locking, but starts with an unlocked database and ends with
an unlocked database. NOTE: at least skiplist and twoskip implement
a hack where if the database IS locked for a non-transactional read
request, they will act as if you'd passed the current transaction in
for the NULL case. This is a hack around layering violations and
kind of sucks.
`&NULL` - e.g:
```c
struct txn *tid = NULL;
const char *data = NULL;
size_t datalen = 0;
int r = cyrusdb_fetch(mydb, key, keylen, &data, &datalen, &tid);
```
After calling this, tid will have an opaque value allocated by the
database backend, which must be passed to all further cyrusdb
operations on that database until either `cyrusdb_commit()` or
`cyrusdb_abort()` are called.
`&tid` - e.g.
```c
if (r == CYRUSDB_NOTFOUND) {
r = cyrusdb_store(mydb, key, keylen, "DEFAULT", 7, &tid); // set a default value
}
```
Given an existing transaction, perform this call in the context of the transaction.
OK, foreach. Foreach is very tricky, because it takes TWO callbacks. The callbacks have an identical signature, but different return codes!
```c
typedef int foreach_p(void *rock,
const char *key, size_t keylen,
const char *data, size_t datalen);
typedef int foreach_cb(void *rock,
const char *key, size_t keylen,
const char *data, size_t datalen);
```
The difference is this: `foreach_p` is called with the database
locked, **always** - even if called without a transaction.
`foreach_p` returns `1` or `0`. `0` means *"skip this record"*, `1`
means *"process this record"*. This is useful to pre-filter records
when called without a transaction, because otherwise you lock and
unlock all the time.
`NULL` for `foreach_p` is treated like a test which always returns `1`, so you can pass `NULL` if you don't need filtering.
If `foreach_p` returns `1`, then with an unlocked transaction, the
database is now unlocked BEFORE calling `foreach_cb`, the callback.
`foreach_cb` returns a `CYRUSDB_` response. If zero, the foreach will
continue. If non-zero, the `foreach` will abort and return the
non-zero response. This is both useful for error cases, and useful
for short-circuiting, if you only care that a key exists, you can do
something like:
```c
static int exists_cb(void *rock __attribute__((unused)), [...])
{
return CYRUSDB_DONE; /* one is enough */
}
```
and then use `exists_cb` as your `foreach_cb` and check if the return code is `CYRUSDB_DONE` to know if the foreach found a key.
If `foreach` is called with a transaction pointer, then it is your responsibility as the caller to also pass that pointer (and a pointer to the database) in that rock, so that callees can make further operations within the same transaction. A foreach with a transaction does NOT unlock before calling its callback.
### Writing
There are three write operations:
- [`cyrusdb_create`](https://github.com/cyrusimap/cyrus-imapd/blob/master/lib/cyrusdb.c#L258)
- [`cyrusdb_store`](https://github.com/cyrusimap/cyrus-imapd/blob/master/lib/cyrusdb.c#L268)
- [`cyrusdb_delete`](https://github.com/cyrusimap/cyrus-imapd/blob/master/lib/cyrusdb.c#L278)
`cyrusdb_store` will either create or overwrite an existing key.
`cyrusdb_create` will abort if the key already exists.
`cyrusdb_delete` takes a flag `force'`which just makes it return
`CYRUSDB_OK (0)` rather than `CYRUSDB_NOTFOUND` if they key doesn't
exist. Strangely, `force` is after `&tid`, making it the only cyrusdb
API that does that, but hey - keeps you on your toes.
`&tid` behaves exactly the same for the write APIs. If not passed, then the database engine will behave as it if creates a writable transaction, does the operation, then commits all within the `cyrusdb_*` call.
### Gotchas!
* `NULL` is permitted in both keys and values, though `flat` and `quotalegacy` have 8-bit cleanliness issues.
* zero-length keys are not supported.
* zero-length values are theoretically supported, but a little
interesting. Certainly, pass "" rather than `NULL` as the value when
writing or things will get weird. I'm pretty sure at least the *skip
databases assert on these kinds of weirdness.
* unlocked `foreach`: this is the land of the gotcha! They key and
data pointers (`const char *`) passed to your `foreach_cb` are only
valid **UNTIL YOU TOUCH THE DATABASE**. A common cause of rare and
hard to diagnose bugs is writing something to the same database in the
same process (**OR EVEN READING FROM IT AGAIN**). I cannot emphasise
this enough. If you want to zero-copy access that data, you need to
access it first, before touching that DB again. Otherwise the map in
which the data was a pointer may have been replaced as the next read
found a new file and mapped it in!
Also: if you're implementing a backend. Unlocked `foreach` must find future records created by the current callback. Consider a database containing 4 keys:
```
A B C D
```
if you are at key `B` and insert a key `BB`, then it must be iterated over. If you insert `AA` while at `B`, it must **NOT** be iterated over.
* Opening the same database multiple times. In the bad old days,
opening the same database multiple times in the same process led to
locking bugs (`fcntl` is braindead). Each database engine is
responsible for making sure this doesn't happen. Most engines keep
a linked lists of open databases. If you try to open the same
database again, they will just return the existing opened copy and
bump a refcount. Beware. If a database is locked and you try to
lock again - thinking you were opening it brand new, it will
assertion fail and/or error.
I think that covers about everything!
Cyrusdb is used just about everywhere that sorted key-value databases
give what's needed, including `mailboxes.db`, `annotations.db` (global and
per mailbox databases), seen state (non-owner), subscriptions,
`cyrus.indexed.db` for `Xapian`, and the rather massive (and increasingly
inaccurately named) `user.conversations`.
Future plans are to increase the usage of cyrusdb databases, possibly
by building an indexing layer on top and using that instead of the
sqldb interface used for sqlite databases by DAV code, and possibly
also moving other custom file formats into a cyrusdb to allow easier
stateless server builds on a distributed backend.
diff --git a/doc/README.xapian b/doc/README.xapian
index cae935c03..4d7d9aead 100644
--- a/doc/README.xapian
+++ b/doc/README.xapian
@@ -1,64 +1,64 @@
How to install Xapian for Cyrus 3.0 or later
============================================
Please follow the installation guide for Cyrus dependencies in the
main README. Then see the section for Xapian configuration.
How to install Xapian for Cyrus 2.5
===================================
To build xapian support into Cyrus, you need to build your own xapian
library as follows:
# choose your own adventures here
export XAPIAN_DIR=/opt/xapian
export CYRUS_DIR=/opt/cyrus
export SRC_DIR=/opt/src
cd $SRC_DIR
wget http://www.cyrusimap.org/releases/cyrus-imapd-3.0.0-beta1.tar.gz
wget http://oligarchy.co.uk/xapian/1.2.21/xapian-core-1.2.21.tar.xz
tar -xf xapian-core-1.2.21.tar.xz
cd xapian-core-1.2.21
tar -xf $cyrusdir/contrib/xapian_quilt.tar.gz
QUILT_PATCHES=xapian_quilt quilt push -a
autoreconf -v -i
./configure --prefix=$XAPIAN_DIR
make
make install
cd $SRC_DIR
tar -xf cyrus-imapd-3.0.0-beta1.tar.gz
cd cyrus-imapd-3.0.0-beta1
XAPIAN_CONFIG=$XAPIAN_DIR/bin/xapian-config ./configure --enable-xapian --prefix=$CYRUS_DIR
make
make install
And you have a Xapian-supporting Cyrus IMAPd!
How to configure Cyrus to use Xapian search
===========================================
You'll need to set up sync log to a channel called squatter, and
at least one search tier.
search_engine: xapian
search_index_headers: no
search_batchsize: 8192
defaultpartition: default
defaultsearchtier: t1
partition-default: /var/cyrus/spool
t1searchpartition-default: /var/cyrus/search
And then you'll need to arrange for a rolling squatter to run on
startup:
cyrus.conf:
DAEMON {
# run a rolling squatter
squatter cmd="squatter -R"
}
If you want to do more complex search tiers and repacking and stuff,
you'll probably want to read:
http://lists.tartarus.org/pipermail/xapian-discuss/2014-October/009112.html
-And see how we do it at FastMail.
+And see how we do it at Fastmail.
diff --git a/doc/legacy/install-configure.html b/doc/legacy/install-configure.html
index ce7413921..71198b2ab 100644
--- a/doc/legacy/install-configure.html
+++ b/doc/legacy/install-configure.html
@@ -1,575 +1,575 @@
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<HTML>
<HEAD>
<meta http-equiv="Content-type" content="text/html;charset=UTF-8">
<TITLE>Installing and configuring the IMAP Server
</title>
</head>
<body>
<h1>Installing and configuring the IMAP Server</h1>
This section describes the shell scripts to run and the configuration
files to modify once "<tt>configure</tt>" and "<tt>make</tt>" have run.
<ol> <li>Create a user and group for the Cyrus subsystem. The
examples in this document assume a user of "<tt>cyrus</tt>" and a
group of "<tt>mail</tt>", though any user and group name can be used.
If a user other than "<tt>cyrus</tt>" is to be used, it must have been
previously specified in the "<TT>--with-cyrus-user=</TT>" option to
"<TT>configure</TT>". If a group other than "<tt>mail</tt>" is to be
used, it must have been previously specified in the
"<TT>--with-cyrus-group=</TT>" option to "<TT>configure</TT>".
<P>
<li>After you've logged in as "<TT>root</TT>", install the cyrus software.
<pre>
<kbd> make install
</kbd></pre>
Be sure that the server programs ended up in the directory specified
by "<tt>--with-cyrus-prefix</tt>" (by default, "<tt>/usr/cyrus/bin</tt>").
<p><li>The Cyrus IMAP server uses the 4.3BSD syslog that separates
messages into both levels and categories. Invoke "<kbd>man
syslog</kbd>" to see if "<tt>openlog()</tt>" takes three arguments. If
it does not, replace the system "<tt>syslogd</tt>" and
"<tt>syslog.conf</tt>" with the files provided in the
"<tt>syslog</tt>" directory.
<pre>
<kbd> mv syslogd /etc/syslogd
mv syslog.conf /etc/syslog.conf
</kbd></pre>
If you do not copy the "<tt>syslog/syslog.conf</tt>" file to the
"<tt>/etc</tt>" directory, be sure to add support for
"<tt>local6.debug</tt>". The file should include a line like:
<pre>
local6.debug /var/log/imapd.log
</pre>
You probably also want to log SASL messages with a line like:
<pre>
auth.debug /var/log/auth.log
</pre>
After installation and testing, you probably want to change the
".debug" component to something a little less verbose. Create the log
files:
<pre>
<kbd> touch /var/log/imapd.log /var/log/auth.log
</kbd></pre>
<li>Create the file "<tt>/etc/imapd.conf</tt>". Here is a sample
"<tt>imapd.conf</tt>" with a minimal number of values defined:
<pre>
configdirectory: /var/imap
partition-default: /var/spool/imap
admins: curtj abell
sasl_pwcheck_method: saslauthd
</pre>
For a description of all the fields in this file, see the
<tt>imapd.conf(5)</tt> man page. (Note
that this file also exports values to libsasl, the most important of
them the <tt>pwcheck_method</tt>. In this example, users are
authenticated via the <tt>saslauthd</tt> daemon, which can be run
in a number of different ways.)
<p><b>READ THE <tt>imapd.conf(5)</tt> MAN PAGE</b>. There are options
in there that you will want to know about and default behavior that
you may not like.
<P>
Note that <b>everyday users should not be administrators</b>. Admins
have powers not granted to regular users and while the server allows
them to receive mail, some problems will occur if admins are used as
regular users. You also should <b>not</b> read mail as an
administrator. You should have separate accounts for reading mail and
administrating. This is especially true if using the
<tt>altnamespace</tt> option, because admins are <b>always</b>
presented with the standard (internal) namespace.
<p><li>Create the configuration directory specified by the
"<tt>configdirectory</tt>" option in "<tt>imapd.conf.</tt>" The
configuration directory is similar in concept to the
"<tt>/usr/lib/news</tt>" directory. It stores information about the
IMAP server as a whole.
<p>This document uses the configuration directory "<tt>/var/imap</tt>"
in its examples. This directory should be owned by the
cyrus user and group and should not permit access to other users.
<pre>
<kbd> cd /var
mkdir imap
chown cyrus imap
chgrp mail imap
chmod 750 imap
</kbd></pre>
<li>Create the default partition directories specified in the
"<tt>/etc/imapd.conf</tt>" file.
<p>This document uses a default partition directory of
"<tt>/var/spool/imap</tt>" in the following example:
<pre>
<kbd> cd /var/spool
mkdir imap
chown cyrus imap
chgrp mail imap
chmod 750 imap
</kbd></pre>
The partition directory is similar in concept to
<tt>/var/spool/news</tt>. It is where the mailboxes are stored.
Unlike most netnews systems, Cyrus allows you to have more
than one partition.
<li>If you wish to use Sieve, and you didn't configure deliver to look
in home directories (see the <tt>imapd.conf</tt> man page), create the
Sieve directory:
<pre>
<kbd> cd /usr
mkdir sieve
chown cyrus sieve
chgrp mail sieve
chmod 750 sieve
</kbd></pre>
<li>Change to the Cyrus user and use the tool
"<tt>tools/mkimap</tt>" to create the rest of the directories
(subdirectories of the directories you just created).
<pre>
<kbd> su cyrus
tools/mkimap
exit
</kbd>
</pre>
If Perl is not available, it should be easy (but time consuming) to
create these directories by hand. <P>
<li><b>LINUX SYSTEMS USING EXT2FS ONLY</b>: Set the user, quota, and
partition directories to update synchronously. Failure to do this may
lead to data corruption and/or loss of mail after a system
crash. Unfortunately, doing so may result in a serious performance
hit. If you are using a newer filesystem than ext2fs on Linux, this
step should not be necessary. (Running ext3 in any mode is safe.)
<pre>
<kbd> cd /var/imap
chattr +S user quota user/* quota/*
chattr +S /var/spool/imap /var/spool/imap/*
</kbd></pre>
Also set the queue directory of the mail daemon to update
synchronously. The following example is for sendmail:
<pre>
<kbd> chattr +S /var/spool/mqueue
</kbd></pre>
<p><li>To enable STARTTLS support, see <a href="#openssl">how to
configure OpenSSL</a> below.
<p><li>Add the following lines to the "<tt>/etc/services</tt>" file if they
aren't already there.
<pre>
pop3 110/tcp
nntp 119/tcp
imap 143/tcp
imsp 406/tcp
nntps 563/tcp
acap 674/tcp
imaps 993/tcp
pop3s 995/tcp
kpop 1109/tcp
lmtp 2003/tcp
sieve 4190/tcp
fud 4201/udp
</pre>
<p><li><b>Remove "<tt>/etc/[x]inetd.conf</tt>" entries.</b> Any
<tt>imap</tt>, <tt>imaps</tt>, <tt>pop3</tt>, <tt>pop3s</tt>,
<tt>kpop</tt>, <tt>lmtp</tt> and <tt>sieve</tt> lines need to be
removed from <tt>/etc/[x]inetd.conf</tt> and [x]inetd needs to be
restarted.
</ol>
<h3><a name="master">Configuring the Master Process</a></h3>
<ol>
<li> Choose a configuration from the <tt>master/conf</tt> directory:
<dl compact>
<dt><tt>small.conf</tt><dd>bare-bones server supporting IMAP and POP
<dt><tt>normal.conf</tt><dd>server supporting IMAP, POP, the SSL
wrapped versions, and the Sieve script management protocol
<dt><tt>prefork.conf</tt><dd>The same configuration as above, but
with some preforked processes for faster processing.
<dt><tt>backend-cmu.conf</tt><dd>Our configuration (for Murder Backend /
typical IMAP servers)
<dt><tt>frontend-cmu.conf</tt><dd>Our configuration (for Murder Frontend servers)
</dl>
<p>To use <tt>normal.conf</tt>, do:
<pre>
<kbd> cp master/conf/normal.conf /etc/cyrus.conf
</kbd></pre>
<p>Optionally, you can edit <tt>/etc/cyrus.conf</tt> to disable or
enable certain services, or to tune the number of preforked copies.
Be sure not to remove the entries that are labeled required.
<p><li>Arrange to start "<tt>/usr/cyrus/bin/master</tt>" as root when
the system starts. It will bind some ports and then give up its root
privileges. Until your system reboots, you can start the master
process by hand:
<pre>
<kbd> /usr/cyrus/bin/master &amp;
</kbd></pre>
<p><li>Monitor the progress of the master process by examining the
<tt>imapd.log</tt> file. It should never exit by itself, but you can
shut down the mail system by sending it a signal with <tt>kill</tt>.
<p><li><b>Clean Shutdown</b> - you can shut the master process down
cleanly by sending it a SIGQUIT rather than SIGTERM signal. This
will cause the master process to send SIGQUIT to all its children and
then wait for them to finish cleanly. This avoids issues like a
message being appended by lmtpd but the sync_log record never being
written.
<p>Since a clean shutdown may never finish if a child process is stuck
for some reason the recommended approach is to send a SIGQUIT then loop
on the master process sending a signal 0 every second until either the
master process has gone away or a suitable time has expired (maybe 10
seconds). You can then send a SIGTERM if the process still exists.
-<p>At FastMail the following snippet of perl is used (warning: Linux
+<p>At Fastmail the following snippet of perl is used (warning: Linux
specific signal numbers - check your own system before using this):
<pre>
my $pid = `cat $PIDFILE`;
chomp($pid);
print "Trying nice shutdown - killing $pid with SIGQUIT\n";
kill 3, $pid;
foreach my $num (1..10) {
if (kill 0, $pid) {
print "Not dead yet after $num seconds\n";
sleep 1;
}
else {
last;
}
}
if (kill 0, $pid) {
print "No more Mr. Nice Guy - killing $pid with SIGTERM\n";
kill 15, $pid;
}
</pre>
</ol>
<h3><a name="mta">Configuring the Mail Transfer Agent</a></h3>
<p>In order to deliver mail to the Cyrus system, you'll have to
configure your MTA (Sendmail, Postfix, Exim, etc.) to use LMTP.
<h4>Configuring <a href="http://www.sendmail.org/">Sendmail</a></h4>
Generate a sendmail configuration file which delivers local mail
to the IMAP server. See the file <TT>cf/README</TT> in the Sendmail
distribution for information on how to create a complete configuration
file. This file also lists variables that can be used to customize
the mailer definitions mentioned below.
<p>The following configurations assume that you are using the
<tt>lmtpunix</tt> service from one of the sample <tt>cyrus.conf</tt>
files discussed above.
<ul>
<li> For Sendmail 8.12.4 and higher, use the <tt>cyrusv2</tt> mailer
definition in the Sendmail distribution:
<pre>
define(`confLOCAL_MAILER', `cyrusv2')
MAILER(`cyrusv2')
</pre>
If you wish to change the name of the UNIX socket or switch to TCP,
define <tt>CYRUSV2_MAILER_ARGS</tt> appropriately as described in
<TT>cf/README</TT>.</li>
<li> For Sendmail 8.10 - 8.12.3, use the <tt>contrib/cyrusv2.mc<tt>
file as a template to create a Sendmail configuration file.</li>
<li> For Sendmail 8.9.x and earlier, use the <tt>cyrus</tt> mailer
definition in the Sendmail distribution:
<pre>
define(`confLOCAL_MAILER', `cyrus')
MAILER(`cyrus')
</pre>
Edit <TT>/etc/group</TT> and
add user "<TT>daemon</TT>" to the "<TT>mail</TT>" group. This will
permit sendmail to run the "<TT>deliver</TT>" (LMTP client) program to
deliver mail to the IMAP server.</li>
</ul>
<p>Cyrus also includes a socket map daemon <tt>smmapd</tt> which can
be used by Sendmail 8.13 and higher (a patch for 8.12 is available) to
verify at RCPT TO time that a message can be delivered to the
particular mailbox. To use this daemon, add <tt>smmapd</tt> as a
service in <tt>cyrus.conf</tt> and configure Sendmail accordingly.
<h4>Configuring <a href="http://www.postfix.org/">Postfix</a></h4>
The Postfix source distribution comes with the
file "<tt>README_FILES/LMTP_README</tt>". Even if you are using a
binary distribution of Postfix, it would be well worth your while
to download the full Postfix source. Not only will you get the
above file, but numerous other "readme" files and sample
configuration files.
<p>One thing you need to watch out for is the UID and GID of
the Postfix software. As it states in the Postfix "<tt>INSTALL</tt>"
document, you must create a new account that does not share its UID
and GID with any other user account. This is for security reasons.
If you installed Postfix with a GID of "<tt>mail</tt>", you will need
to select a different GID for Cyrus. See the Cyrus configure options
"<tt>--with-cyrus-user</tt>" and "<tt>--with-cyrus-group</tt>". (This
was more crucial when the use of Cyrus' "<tt>deliver</tt>" was more
prevalent, but it is still a good idea to follow this policy.)
<p>Another thing to note is the location of your "<tt>sendmail</tt>"
command. On some platforms this will be "<tt>/usr/sbin/sendmail</tt>",
on others, "<tt>/usr/lib/sendmail</tt>". Cyrus will need to know where
this command is. See <a href="install-sieve.html">Installing Sieve</a>
for more details.
<p>Assuming that you are using the <tt>lmtpunix</tt> service from one
of the sample <tt>cyrus.conf</tt> files discussed above, the Postfix
configuration file "<tt>/etc/postfix/main.cf</tt>" should have the
following line:
<pre>
mailbox_transport = lmtp:unix:/var/imap/socket/lmtp
</pre>
<p>Naturally, both the Postfix UID and the Cyrus UID need to be
able to access the specified socket file.
<p>Starting with Postfix snapshot-20010222, you can improve the
efficiency of LMTP delivery via the "<tt>mailbox_transport</tt>" by
putting the following entries in "<tt>/etc/postfix/main.cf</tt>":
<pre>
local_destination_recipient_limit = 300
local_destination_concurrency_limit = 5
</pre>
<p>Of course you should adjust these settings as appropriate for
the capacity of the hardware you are using. The recipient limit
setting can be used to take advantage of the single instance
message store capability of Cyrus. The concurrency limit can be
used to control how many simultaneous LMTP sessions will be
permitted to the Cyrus message store.
<p>Additional examples are included in the Postfix
file "<tt>README_FILES/LMTP_README</tt>".
<h4>Configuring <a href="http://www.exim.org/">Exim 4</a></h4>
<p>Generate an Exim configuration file which delivers local mail to the
IMAP server. See the Exim documentation for information on how to
create a complete configuration file.
<p>Cyrus is designed to be used as a black-box server -- there
are usually no local user accounts. As a result, you must define the
following "router":
<pre>
localuser:
driver = accept
transport = local_delivery
</pre>
<p>The following "transports" assume that you are using either the
<tt>lmtpunix</tt> or <tt>lmtp</tt> service from one of the sample
<tt>cyrus.conf</tt> files discussed above.
<ul>
<li> Using <tt>lmtpunix</tt> (UNIX socket):
<pre>
local_delivery:
driver = lmtp
command = "/usr/cyrus/bin/deliver -l"
batch_max = 20
user = cyrus
</pre>
</li>
<li> Using <tt>lmtp</tt> (TCP socket -- Exim and Cyrus on same host):
<pre>
local_delivery:
driver = smtp
protocol = lmtp
hosts = localhost
allow_localhost
</pre>
</li>
</ul>
<p>For more advanced configurations (such as address verification, etc),
consult the Exim documentation and sample configurations.
<h3>Exporting Netnews via IMAP</h3>
If you wish to use export Netnews via IMAP, consult <a
href="install-netnews.html">install-netnews.html</a>.
<h2><a name="openssl">SSL, TLS, and OpenSSL</a></h2>
<p>Transport Layer Security (TLS), is a standardized version of the Secure
Sockets Layer (SSL v3) standard. IMAP can make use of two different
versions of TLS/SSL: STARTTLS and an SSL wrapped session.</p>
<p>In STARTTLS, a client connects to the IMAP port as normal and then
issues the STARTTLS command, which begins a TLS negotiation. This is
currently supported by the Cyrus IMAP server when it is compiled with
OpenSSL.</p>
<p>The alternative, a SSL wrapped connection, involves the client
connected to a separate port ("<tt>imaps</tt>") and negotiating a SSL
session before starting the IMAP protocol. Again, this is supported
natively by the Cyrus IMAP server when it is compiled with
OpenSSL.</p>
<p>Both TLS and SSL require a server key and a certificate.
Optionally, in addition to establishing a secure connection, TLS can
authenticate the client.</p>
<h3>Configuring Cyrus with OpenSSL</h3>
<ol>
<li><p>OpenSSL requires the certificate and key in PEM format. You can
create the server's private key and certificate yourself using
OpenSSL. Here, we create a self-signed key for the machine
"<tt>foobar.andrew.cmu.edu</tt>" and put both the certificate and key
in the file "<tt>/var/imap/server.pem</tt>".</p>
<p>Please do not blindly enter in the information to OpenSSL
below. Instead, enter the appropriate information for your
organization (i.e., NOT Carnegie Mellon University for the Organization
Name, etc.).</p>
<pre>
<kbd>openssl req -new -x509 -nodes -out /var/imap/server.pem -keyout /var/imap/server.pem -days 365</kbd>
Using configuration from /usr/local/lib/openssl/openssl.cnf
Generating a 1024 bit RSA private key
.............+++++
......................+++++
writing new private key to '/var/imap/server.pem'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:<kbd>US</kbd>
State or Province Name (full name) [Some-State]:<kbd>Pennsylvania</kbd>
Locality Name (eg, city) []:<kbd>Pittsburgh</kbd>
Organization Name (eg, company) [Internet Widgits Pty Ltd]:<kbd>Carnegie Mellon University</kbd>
Organizational Unit Name (eg, section) []:<kbd>Andrew Systems Group</kbd>
Common Name (eg, YOUR name) []:<kbd>foobar.andrew.cmu.edu</kbd>
Email Address []:
</pre></li>
<li>Make sure to make key file(s) readable by the Cyrus user. For
example: <kbd>chown cyrus /var/imap/server.pem</kbd>
<li>Add the following to <tt>/etc/imapd.conf</tt> to tell the server
where to find the certificate and key files (used for ALL services):
<pre>server_cert_file: /var/imap/server.cert
server_key_file: /var/imap/server.key
</pre>
Optionally, you can use separate certificates and key files for each
service:
<pre>[servicename]_server_cert_file: /var/imap/imap-server.cert
[servicename]_server_key_file: /var/imap/imap-server.key
</pre>
&quot;servicename&quot; here refers to the name of the service as specified
in <tt>cyrus.conf</tt>. It is <i>not</i> necessarily the name of the binary.
<p>This is useful if you want to use different hostnames for each service
(e.g., via virtual host interfaces or DNS CNAMEs). In the absence of
any of the service specific options, the value of the global option is
used. A value of <b>disabled</b> for the certificate or key file for
a particular service will disable SSL/TLS for that service.
<p>If you have a Certificate Authority (CA), you may wish to generate
a certificate request and send it to be signed by your CA.
<p>By default, Cyrus will cache SSL/TLS sessions for reuse for up to
24 hours. By adjusting the value of the <tt>tls_session_timeout</tt> option in
<tt>imapd.conf</tt>, the session caching can be disabled (0) or the
expiration period shortened.
<p>
</li>
<li>You can test STARTTLS by using <tt>imtest</tt>:
<pre><kbd>imtest -t "" foobar.andrew.cmu.edu
</kbd></pre>
</ol>
<h3>Client-side certificates</h3>
<p>Client certificates are somewhat harder to configure than server
certificates. You'll need a CA (certificate authority) and need to
generate client certificates signed by that CA. STARTTLS in Sendmail
and other MTAs have similar problems, so <a
href="http://www.sendmail.org/~ca/email/starttls.html">Claus Assman's
page</a> is a good reference.</p>
<p>You can use the self-signed certificate generated above as a CA for
client certificates. To do this, try the following:</p>
<p><b>TODO:</b> write me!</p>
<p>Unfortunately, there's no standard on how to convert the client's
authenticate DN (distinguished name) to a SASL authentication
name.</p>
<h2><a name="altnamespace">Alternate Namespace and UNIX Hierarchy
Convention</a></h2>
If you wish to use the alternate namespace and/or the UNIX hierarchy
convention, consult <a
href="altnamespace.html#altnameconfig">altnamespace.html</a>.
<P><HR>
last modified: $Date: 2010/01/06 17:01:29 $
</BODY></HTML>

File Metadata

Mime Type
text/x-diff
Expires
Fri, Apr 24, 10:45 AM (3 d, 3 h ago)
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
18881238
Default Alt Text
(38 KB)

Event Timeline