WikiSuite | Cyrus IMAP 3 and JMAP development

Please see: blogpost16-WikiSuite-will-now-support-all-major-Linux-distros. While ClearOS's mail stack is powered by Cyrus IMAP, Virtualmin's mail stack is with Dovecot, so we will focus on efforts there. This page is kept for posterity.

WikiSuite will add support for JMAP via a combination of contributions to Cyrus, Cypht and the IETF standardization process.

Cyrus 3.2 with JMAP support is planned for March 2020.

On July 14 and 15, 2018, WikiSuite team members were at the July 2018 JMAP IETF Hackathon in Montréal , which was a follow-up to the March 17 and 18, 2018 event, where WikiSuite team members joined Fastmail at the IETF 101 Hackathon to work on JMAP. -> 2018 XMPP, WebRTC and JMAP Hackathons in London

As of 2019, the reference implementation is on ClearOS, but the future production implementation will be using containers.

ClearOS's 7.x's mail server is Cyrus IMAP 2.x This page is to coordinate testing of upcoming Cyrus IMAP 3.x , which brings new features such as CalDAV, CardDAV and JMAP support.

Status

A lot of work is happening on the JMAP spec and associated code. We'll update cyrus.suite.wiki in ~~November or December 2017~~ ~~January~~ ~~February 2018~~ before the event to the latest developer snapshot.

Keep informed of JMAP-related work on Cyrus

issues related to JMAP on Github
Weekly updates on the Cyrus-devel Mailing list

Code

https://github.com/cyrusimap/cyrus-imapd (Cyrus)
https://gitlab.com/clearos/upstream/centos/cyrus-imapd/tree/clear7 (Cyrus 2.x app for ClearOS)
https://gitlab.com/clearos/clearfoundation/app-smtp (Postfix app for ClearOS)

Commits

JMAP spec

Recent Commits to jmap:master Make ShareNotification headings consistent calendars: Match sharing language/formatting RFC9670 final version calendars: Publish -21 draft calendars: Use SVG for diagram calendars: Format examples as code calendars: Update example to be more recent calendars: Update formatting to match sharing calendars: Address Éric's review comments Final updates to RFC9670	Recent Commits to cyrus-imapd:master Merge pull request #5124 from dilyanpalauzov/xapian_wrap_doc_objects Merge pull request #5127 from dilyanpalauzov/mboxlist_create_namecheck Merge pull request #5113 from rsto/jmap_email_get_bodypart_type_invalid mboxlist_create_namecheck() memory leak Merge pull request #5123 from elliefm/v311/master-cleanup-after-crash… xapian_wrap: in struct xapian_doc to use objects instead of pointers xapian_snipgen_doc_part() utilize range-based for loop xapian_wrap:write_db_versions() accept versions as const Info: test cleanup of proc entry after service crash master: clean proc entries from crashed services

What is new in Cyrus 3.x

cyrus.suite.wiki

cyrus.suite.wiki is a shared development and testing server. Once the testing period is over, the server will be deleted so don't store any real data on this server. If you would like server access, please contact Marc Laporte

Urls

You can replace test with your login and follow the pattern below

User management

https://cyrus.suite.wiki:81/app/users (you need root access). And users need to login via IMAP for their account to be properly set up.

IMAP and SMTP

IMAP

port 143
server: cyrus.suite.wiki
username: login (not full email)

SMTP

port: 25?
server: cyrus.suite.wiki
username: login (not full email)
not working for me

IMAPS

SSL
port 993
server: cyrus.suite.wiki
username: login (not full email)
not working for me

SMTPS

SSL
port: 587
server: cyrus.suite.wiki
username: login (not full email)
not working for me

JMAP webmail

Test URLS, automatically logged in, as a different user:

http://cyrus.suite.wiki:8080/ (test inbox provided by upstream)
http://cyrus.suite.wiki:8080/user-nikola.html
http://cyrus.suite.wiki:8080/user-test.html

Old non-working attempt

[+]

https://cyrus.suite.wiki:81/jmap-demo-webmail/ (not working yet)

The webmail demo is installed, but it looks like the authentication layer in the client is different than what is expected in Cyrus? I updated the JMAP.auth.didAuthenticate block in /usr/share/jmap-demo-webmail/app/state.js as described in the JMAP webmail demo docs, but I see 401 unauthorized replies.

Here's the URL of the webmail demo - please make sure your adblocker is disabled! https://cyrus.suite.wiki:81/jmap-demo-webmail/

It's installed in /usr/share/jmap-demo-webmail. Use "git diff" to see the minor changes.

[root@cyrus jmap-demo-webmail]# pwd
/usr/share/jmap-demo-webmail
[root@cyrus jmap-demo-webmail]# git diff
diff --git a/app/state.js b/app/state.js
index 82ec7a4..f1b5310 100644
--- a/app/state.js
+++ b/app/state.js
@@ -348,9 +348,9 @@ App.kbshortcuts
 var userPath = ( /k=([0-9a-f\-]+)/.exec( location.href ) || [ '', '' ] )[1];
 
 JMAP.auth.didAuthenticate({
-    username:       'user@example.com',
-    accessToken:    'password',
-    apiUrl:         '/jmap/'   + userPath,
+    username:       'test',
+    accessToken:    'wikisuite',
+    apiUrl:         'https://cyrus.suite.wiki/jmap',
     eventSourceUrl: '/events/' + userPath,
     uploadUrl:      '/upload/' + userPath,
     downloadUrl:    '/raw/'    + userPath + '/{blobId}/{name}'
diff --git a/index.html b/index.html
index 330c622..1c7b9f6 100644
--- a/index.html
+++ b/index.html
@@ -10,7 +10,7 @@
 </head>
 <body>
 <!-- Comment out the following line to talk to a real server -->
-<script type="text/javascript" src="fixtures.js"></script>
+<!-- script type="text/javascript" src="fixtures.js"></script -->
 
 <script type="text/javascript" src="libs/Overture.js"></script>
 <script type="text/javascript" src="libs/timezones-raw.js"></script>
[root@cyrus jmap-demo-webmail]#

Confirmation that JMAP is working

[root@cyrus ~]# curl -X POST -H "Content-Type: application/json" -H "Accept: application/json" --user test:wikisuite -d '[["getCalendars", {}, "#1"]]' http://localhost/jmap/
[["calendars",{"accountId":"test","state":"15","list":[{"id":"Default","x-href":"/dav/calendars/user/test/Default","name":"personal","mayReadFreeBusy":true,"mayReadItems":true,"mayAddItems":true,"mayModifyItems":true,"mayRemoveItems":true,"mayRename":true,"mayDelete":true}],"notFound":null},"#1"]]
[root@cyrus ~]#

[root@cyrus ~]# curl -X POST -H "Content-Type: application/json" -H "Accept: application/json" --user test:wikisuite -d '[[ "getMailboxes", {}, "#0" ]]' http://localhost/jmap/
[["mailboxes",{"accountId":"test","state":"23","notFound":null,"list":[{"id":"23b8b283-54b2-4953-83b1-dddfdf50d1ab","name":"Inbox","mustBeOnlyMailbox":false,"mayReadItems":true,"mayAddItems":true,"mayRemoveItems":true,"mayCreateChild":true,"totalMessages":1,"unreadMessages":0,"totalThreads":1,"unreadThreads":0,"parentId":null,"mayRename":false,"mayDelete":false,"role":"inbox","sortOrder":1},{"id":"5e8d9e35-76a6-4a54-9bb4-e8625f908670","name":"Drafts","mustBeOnlyMailbox":false,"mayReadItems":true,"mayAddItems":true,"mayRemoveItems":true,"mayCreateChild":true,"totalMessages":1,"unreadMessages":1,"totalThreads":1,"unreadThreads":0,"parentId":null,"mayRename":true,"mayDelete":true,"role":null,"sortOrder":10},{"id":"3334d6fb-f621-4b7b-9634-93eb193023d2","name":"Trash","mustBeOnlyMailbox":false,"mayReadItems":true,"mayAddItems":true,"mayRemoveItems":true,"mayCreateChild":true,"totalMessages":0,"unreadMessages":0,"totalThreads":0,"unreadThreads":0,"parentId":null,"mayRename":true,"mayDelete":true,"role":null,"sortOrder":10}]},"#0"]]
[root@cyrus ~]#

Notes for future documentation page

How to install

Set up MX records
install the eGloo release package: rpm -Uvh http://mirrors.egloo.ca/egloo/clearos/7.3/stable/x86_64/egloo-release-7-2.noarch.rpm
- This will be simpler in the future

Then install the IMAP and SMTP apps:

yum --enablerepo=egloo-clearos-devel install app-imap app-smtp

You will also need to do the following:
- Create the default SSL certificate via the “System - Settings - Certificate Manager” app.
- Create a user with IMAP and SMTP enabled.
- Login via an IMAP client in order to create a mailbox for the user.

Optional: How to install Mail Forwarding on ClearOS

How to use

The extra features (CalDAV, CardDAV and JMAP) run on port 80 and/or 443, so Cyrus IMAP interferes with web server. On this dev server, we use port 8080 for the JMAP client
See URLs patterns above

Clients tested

DAVdroid

Automatic detection from email doesn't work but entering username, password and base URL works

Questions

Is it possible to have a logout feature? (which would make it easier to test with different accounts)

Known bugs / limitations

Accounts created in ClearOS are not reliably setup for CalDAV and CardDAV.

Solution: login to the account via IMAP.

peter.baldwin wrote:

Mailbox creation gotcha: The mailboxes get created on the fly:
- the first time the user logs in, or
- a mail message is delivered

peter.baldwin wrote:

The mailboxes for test and nikola.tesla were created, but the relevant subfolders were not. You can browse the mailboxes in /var/spool/imap/ by the way. Here's what it looked like in the log:

Aug 18 14:52:16 cyrus lmtp[12030]: autocreateinbox: autocreate inbox for user test was called
Aug 18 14:52:16 cyrus lmtp[12030]: autocreateinbox: User test, INBOX was successfully created
Aug 18 14:52:16 cyrus lmtp[12030]: WARNING: sieve script /var/lib/imap/sieve/t/test/defaultbc doesn't exist: No such file or directory
Aug 18 14:52:16 cyrus lmtp[12030]: Delivered: to mailbox: user.test

Looking back at my "peter" account, I can see that tried the RSS feed link for a sanity check - https://cyrus.suite.wiki/rss/user/peter This prompted me for a login and I could view my RSS feed. Under the hood, this showed up in the log:

Aug 17 18:18:47 cyrus https[19262]: login: cpe.teksavvy.com peter Basic+TLS User logged in SESSIONID=
...
Aug 17 18:18:47 cyrus https[19262]: creating sql_db /var/lib/imap/user/p/peter.dav

It looks like that "dav" folder is needed by the calendars and addressbook but only created on an IMAP login. If you were to login using an IMAP client, I'm guessing everything would be ok.