Cyrus IMAP 2.5.0 Release Notes
******************************

Important:

  Make sure to read the Upgrading to Cyrus IMAP 2.5.0 notes (all of
  them).

* New Features

* Configuration Option Changes and Enhancements

* Development: Switch to autoconf and libtool

* Development: Switch to Phabricator

* Development: Sphinx for Documentation

Download from GitHub:

   * https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-
     imapd-2.5.0/cyrus-imapd-2.5.0.tar.gz

   * https://github.com/cyrusimap/cyrus-imapd/releases/download/cyrus-
     imapd-2.5.0/cyrus-imapd-2.5.0.tar.gz.sig


Upgrading to Cyrus IMAP 2.5.0
=============================

It is strongly recommended to shut down the Cyrus IMAP services before
performing the upgrade, as the newer binaries will end up writing to
"mailboxes.db" in a way that is not compatible with the older binaries
(that would otherwise still be running).

You can start the server immediately after upgrading, however.


Underscores in "cmd" Names in cyrus.conf(5)
-------------------------------------------

Underscores (the "_" character) are no longer allowed in the "START",
"SERVICES" and "EVENTS" sections of cyrus.conf(5), as they interfere
with configuration options in imapd.conf(5) being prefixed by service
names and an underscore ("_") character.


Database Format Upgrade for Mailboxes
-------------------------------------

Upgrading to Cyrus IMAP 2.5.0 recommends the upgrade of database
formats, for which the reconstruct utility has been modified allowing
administrators to run:

   # **/usr/lib/cyrus-imapd/reconstruct -V max** [options]

This command can be run at the administrator's convenience, and while
running the database format upgrade process is not strictly required,
it is certainly strongly recommended.

Releases prior to 2.5.0, namely the 2.4 series, upgraded "cyrus.index"
database formats in-line (i.e. as the mailbox in question as being
used). This may have caused an I/O storm in some situations, which
2.5.0 aims to address.

The minor version of the "cyrus.index" file in 2.5.0 is **13**. While
mailboxes with an older format will continue to work "forever", newer
features such as annotation quotas will fail, as there is no space in
the database formats prior to version 13 to store the new feature's
information.

Until you upgrade the database format, you may experience slightly
inconsistent search results, due to the "cyrus.cache" format in Cyrus
IMAP versions prior to 2.4.0 running words together and not obeying
word boundaries (as well as versions in the 2.4 and 2.5 series).

If you were running Cyrus 2.3.x with "expunge_mode: delayed", then the
contents of the "cyrus.expunge" file will be removed the first time
the mailbox is opened, and the mailbox will act as if "expunge_mode"
was set to "immediate" -- until it is upgraded.

This is because version 2.3 mailboxes used the "cyrus.expunge" file,
and it's not worth the complexity to try to recreate that file.

This upgrade path supports upgrades from Cyrus IMAP version 2.2.12 and
later versions.

Important:

  The above command can run while the Cyrus IMAP services are in
  operation, and does **not** interfere with the service's operations.

Note:

  Additional options can be specified to reconstruct(8) to more
  granularly specify which mailboxes' "cyrus.index" is to be upgraded.


Quota Fixes and Enhancements
----------------------------

After all mailboxes are upgraded, you should run "quota -f" to make
all message and folder counts are correct. The new quota types that
Cyrus IMAP 2.5.0 supports require the counts to need to be accurate.


Cyrus IMAP Murder Topologies
----------------------------

Environments that run a Cyrus IMAP Murder topology will want to
upgrade their backends before they upgrade their frontends.


Greater Memory Footprint
------------------------

For those upgrading from 2.3.X; newer releases of Cyrus IMAP will use
significantly more memory per selected mailbox.  This is not an error
or bug; it's a feature.  The newer code is holding more data and
metadata in memory for purposes of faster access to more of the
mailbox.  This is not a memory leak.


New Features
============


Index Namelock Release
----------------------

Long-running (idling) connections may have previously intervened with
mailbox deletions and general cleanup work, resulting in one
connection blocking another from, for example, deleting and recreating
a mailbox:

   C1: SELECT "Foo"
   C2: DELETE "Foo"
   C2: CREATE "Foo"

The "CREATE" command on line 3 (by client 2, while client 1 still has
the "Foo" mailbox selected) would have failed before, but does now
succeed.


Extended Quota Types
--------------------

New ways to restrict resource usage:

* Number of Folders,

* Number of Messages,

* Number of Annotations


CalDAV and CardDAV Support
--------------------------

CalDAV and CardDAV support no longer live out-of-tree in the same GIT
repository, but are now mainstream and included in Cyrus IMAP 2.5.0.

This has been a major effort by Carnegie Mellon University and
FastMail, and continues to be, enriching the experience of Cyrus IMAP
users globally.

If upgrading to Cyrus IMAP 2.5 from one of the 2.4.17-caldav-beta
releases, you MUST run the "dav_reconstruct" utility for each of your
CalDAV users.


Support for RFC 5464: IMAP METADATA
-----------------------------------

Cyrus IMAP now fully supports **RFC 5464**, *The IMAP METADATA
Extension*.

This also means the support for the ANNOTATEMORE draft for IMAP will
ultimately be dropped.

Aside from the trusted folder metadata, this also introduces message
annotations. Users will need to be given the "n" right to allow them
to set message annotations.


Event Notifications
-------------------

Various events occuring in Cyrus IMAP, among which mailbox, message
and access events, can now be pushed out through a side-channel and
notify client applications or provide other infrastructure with
detailed information.


Mailbox Distribution Enhancements: Backend and Partition Selection
------------------------------------------------------------------

Thanks to the work of Julien Coloos and colleagues, a new mode is
available for server and partition selection upon mailbox creation.

Prior to Cyrus IMAP 2.5.0, the server and/or partition on which to
create a new mailbox was selected by detecting the largest amount of
absolute free disk space on all servers and partitions. The mailbox
distribution feature allows for more intelligent and flexible routines
to be used in the selection. Please see our Administrator Guide for
more details.


New Database Format for "mailboxes.db"
--------------------------------------

The database format for mailboxes.db has been upgraded, adding;

* A new mailbox type for deleted mailboxes.

  In versions of Cyrus IMAP prior to 2.5.0, mailboxes that were
  deleted may have become unavailable for actual cleanup expecting
  another session on the same mailbox to clean up the directories and
  files. See also Index Namelock Release.

* A key-value storage format is used, allowing for faster and better
  parsing of "mailboxes.db", more granular updates to runtime
  environments, and more sustainable future upgrades.


New Database Format "twoskip"
-----------------------------

A new database format has been added, called "twoskip" [1].

This new database format is reputedly better, faster, safer and 64-bit
capable, as opposed to our former favorite "skiplist".

"twoskip" can be used for the following databases:

* "annotation_db"

* "duplicate_db"

* "mboxkey_db"

* "mboxlist_db"

* "ptscache_db"

* "quota_db"

* "seenstate_db"

* "subscription_db"

* "statuscache_db"

* "tls_sessions_db"

* "user_deny_db"


Miscellaneous
-------------


Allowing Undefined Annotations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Cyrus IMAP 2.5.0 allows administrators to configure that undefined
annotations should be allowed, using a new
"annotation_allow_undefined" setting in imapd.conf(5).


Catchall Mailbox for LMTP
~~~~~~~~~~~~~~~~~~~~~~~~~

Thanks to the work by Carsten Hoeger and Ralf Haferkamp, this new
feature enables administrators to configure a target mailbox for mail
delivered through LMTP to targetted mailboxes that do not exist.

For example, a mail that LMTP would deliver to "user/bovik", which for
the sake of argument does not exist in this example, setting
"lmtp_catchall_mailbox" to "admin" will instead deliver the mail to
"user/admin".

Note:

  **Mailbox name, not Email Address**Note that
  **lmtp_catchall_mailbox** must be a user mailbox name, not an email
  address. Also note that the **user/** namespace indicator as well as
  the hierarchy separator are to be omitted.


Does this impact lmtp_fuzzy_mailbox_match?
""""""""""""""""""""""""""""""""""""""""""

Environments that have "lmtp_fuzzy_mailbox_match" enabled, in order to
have LMTP seek from the targetted, non-existent mailbox sub-folder
(example: "user/bovik/spam/probably") all the way to the toplevel
mailbox folder (i.e. "user/bovik") until it finds a mailbox
(sub-)folder that does exist (example: "user/bovik/spam"), are not
impacted by this setting.


Can the lmtp_catchall_mailbox include the path to a sub-folder of a target mailbox?
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

UNCONFIRMED


Can the lmtp_catchall_mailbox be a shared folder?
"""""""""""""""""""""""""""""""""""""""""""""""""

UNCONFIRMED


Callout for SETMETADATA
~~~~~~~~~~~~~~~~~~~~~~~

A callout program can be called when annotations are set, configured
through "annotation_callout".


Host & User Login Restrictions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Logins can now be restricted on a per host (source IP address) or per
user basis, using the settings "maxlogins_per_host" and
"maxlogins_per_user".


Configuration Option Changes and Enhancements
=============================================

Important:

  While it is not mandatory to update your configuration file with
  these new settings, not doing so may have undesired side-effects,
  including but not limited to deprecation warnings in log messages.


Option Name Changes for "autocreate"
------------------------------------

The options related to automatic creation of user mailboxes and sub-
folders (aka. *autocreate*) have been changed to hold a prefix of
"autocreate_".

The following *autocreate* options are now available:

**autocreate_inbox_folders** (was: "autocreateinboxfolders")

   "autocreate_inbox_folders" controls which folders to create in
   addition to the INBOX folder.

   Separate the folder names by "|".

**autocreate_post** (was: "createonpost")

   Controls whether or not to create a folder when a message is first
   posted to it (by LTMP).

**autocreate_quota** (was: "autocreatequota")

   When creating a user mailbox, set the quota for that mailbox to the
   value of this configuration option.

**autocreate_quota_messages** (not available)

   When creating a user mailbox, set the message quota (maximum number
   of messages allowed in the folder hierarchy) to the value of this
   configuration option.

**autocreate_sieve_folders** (was: "autosievefolders")

   Limit the folders that can be created automatically by a Sieve
   script performing a "fileinto" action, to the folders listed in
   this configuration option.

   Separate the folder names by "|".

**autocreate_sieve_script** (unchanged)

   When creating a user mailbox, associate the Sieve script configured
   here.

**autocreate_sieve_script_compile** (was:
"generate_compiled_sieve_script")

   Whether or not to compile the Sieve script configured by
   "autocreate_sieve_script".

**autocreate_sieve_script_compiled** (was:
"autocreate_sieve_compiled_script")

   When creating a user mailbox, associate the already compiled Sieve
   script configured here.

**autocreate_subscribe_folders** (was: "autosubscribeinboxfolders")

   List the folder names to which the user for which a mailbox is
   being created should be subscribed.

   Note:

     All folders listed here are considered to reside in the personal
     namespace.

   Separate the folder names by "|".

**autocreate_subscribe_sharedfolders** (was:
"autosubscribesharedfolders")

   List the folder names of shared folders to which the user for which
   a mailbox is being automatically created should be subscribed.

   Separate the folder names by "|".

**autocreate_subscribe_sharedfolders_all** (was:
"autosubscribe_all_sharedfolders")

   Rather than subscribe the user for which a mailbox is being
   automatically created to some shared folders, simply subscribe the
   user to all shared folders.

**autocreate_users** (unchanged)

   Limit the users for which mailboxes may be created to the list
   configured here.


Default Change: "delete_mode"
-----------------------------

The default for the imapd.conf(5) configuration option "delete_mode"
has changed from "immediate" to "delayed".

This causes mail folders that are deleted by a client to not
immediately dissappear from the filesystem. Instead, they are renamed
to a deleted namespace that is visible only to administrators.

A separate job "cyr_expire -D $x" is to be included in the master
service configuration file cyrus.conf(5), specifically in the EVENTS
section. "$x" is a number of days to keep already deleted folders.

**Example section of :cyrusman:`cyrus.conf(5)`**

      EVENTS {
          deleteprune cmd="cyr_expire -D 69" at=0430
      }

In the aforementioned example, folders are purged from the filesystem
only after 2 times 31 plus 7 days, corresponding with 2 cycles of a
monthly (full, virtual) backup of which one might fail.


Default Change: "expunge_mode"
------------------------------

The default for the imapd.conf(5) configuration option "expunge_mode"
has changed from "default" to "delayed".

This causes the mail message files associated with messages that are
flagged as Deleted in a folder that is subsequently expunged, or
individual messages that are expunged, to not be removed from the
filesystem directly.

A separate job "cyr_expire -X $x" is to be included in the master
service configuration file cyrus.conf(5), specifically in the EVENTS
section. "$x" is a number of days to keep the message files on the
filesystem.

**Example section of :cyrusman:`cyrus.conf(5)`**

      EVENTS {
          expungeprune cmd="cyr_expire -X 69" at=0430
      }

In the aforementioned example, message files are purged from the
filesystem only after 2 times 31 plus 7 days, corresponding with 2
cycles of a monthly (full, virtual) backup of which one might fail.


Option Name Changes for "ldap_tls_*"
------------------------------------

Configuration option names for LDAP SSL/TLS configuration in
imapd.conf(5) have been changed:

**ldap_ca_dir** (was: "ldap_tls_cacert_dir")

**ldap_ca_file** (was: "ldap_tls_cacert_file")

**ldap_client_cert** (was: "ldap_tls_cert")

**ldap_verify_peer** (was: "ldap_tls_check_peer")

**ldap_ciphers** (was: "ldap_tls_ciphers")

**ldap_client_key** (was: "ldap_tls_key")


Option Name Changes for "tls_*"
-------------------------------

Configuration option names for SSL/TLS configuration in imapd.conf(5)
have been changed to better reflect how they are used, as enhancements
would otherwise create great confusion.

**tls_client_ca_dir** (was: "tls_ca_path")

**tls_client_ca_file** (was: "tls_ca_file")

   The former "tls_ca_*" configuration options specified one or more
   SSL Certificate Authority certificates against which SSL
   certificates offered by clients could be verified.

   In a Cyrus IMAP Murder topology however, Cyrus IMAP servers
   themselves become clients of other Cyrus IMAP servers, but may not
   have been issued certificates under the same verification chain.

With the (too) generic names for "tls_ca_*" configuration options out
of the way, Cyrus IMAP 2.5.0 adds the following configuration options:

**tls_server_cert** (was: "tls_cert_file")

**tls_server_key** (was: "tls_key_file")

   Server SSL certificate and key to use for connections from clients.

**tls_sessions_db** (was: "tlscache_db")

**tls_sessions_db_path** (was: "tlscache_db_path")


New Options for "tls_*"
-----------------------

**tls_client_cert** (<none>)

**tls_client_key** (<none>)

   Client SSL certificate and key to use when cyrus-imapd behaves as a
   client (to other cyrus-imapd server (instances)).

**tls_client_ca_file** (<none>)

**tls_client_ca_dir** (<none>)

   Certificate Authority file or directory used to verify client SSL
   certificates.

**tls_client_certs** ("optional")

   Disable ("off"), allow ("optional") or require ("require") clients
   authenticate with an SSL certificate.

**tls_server_ca_file** (<none>)

**tls_server_ca_dir** (<none>)

   Certificate Authority file or directory used to verify SSL
   certificates offered by other servers.

**tls_compression** ("0")

   Enable TLS compression. Disabled by default.

**tls_eccurve** ("prime256v1")

   Select the elliptic curve used for ECDHE. See **openssl ecparams
   -list_curves** for supported values on your platform.

**tls_prefer_server_ciphers** ("0")

   Prefer the cipher order configured on the server-side.

**tls_versions** ("ssl2 ssl3 tls1_0 tls1_1 tls1_2")

   Disable SSL/TLS protocols not in this list.


Development: Switch to "autoconf" and "libtool"
===============================================

With the release of Cyrus IMAP 2.5.0, the Cyrus IMAP project has
switched to using autoconf and libtool.


Development: Switch to Phabricator
==================================

An instance of Phabricator is going to be replacing our old Bugzilla.

We believe this better facilitates our processes, and will make it
easier to contribute code and collaborate.

Please see https://git.cyrus.foundation/.

Note:

  The Phabricator installation suffered a disk failure. We now use
  GitHub for code and bug management. https://github.com/cyrusimap
  /cyrus-imapd


Development: Sphinx for Documentation
=====================================

While a work in progress still, you're looking at the new and improved
documentation effort for the Cyrus project as a whole.

This documentation is written in reStructuredText, and rendered by
Sphinx.

The GIT repository for the documentation is at

-[ Footnotes ]-

[1] http://opera.brong.fastmail.fm.user.fm/talks/twoskip/twoskip-
    yapc12.pdf
