MS Exchange (and Kerio) auto-discovery

In a previous post, I talked about how CalDAV clients can use auto-discovery to find the account and calendar resources (collections). In this post, I will explain how it's done for clients that supports Exchange Web Services (EWS).

The whole auto discovery process is documented on Microsoft TechNet, but I will give a summary, in a EWS context (it looks like the process for Active Directory/Outlook on Windows process in a bit different). This process applies both to MS Exchange and Kerio Connect.

As with CalDAV, the auto discovery process will take the domain from email address of the client, and it will try to make a HTTP POST to either:

• https://autodiscover.mydomain.com/autodiscover/autodiscover.xml
• http://autodiscover.mydomain.com/autodiscover/autodiscover.xml

For example, if your email address is probert@conatus.lan, it will try to do the POST request at https://autodiscover.conatus.lan/autodiscover/autodiscover.xml, and if it didn't get a valid response, it will try the non-secure port.

If the autodiscover.mydomain.com DNS entry is not find, the client will try to do the POST request to https://mydomain.com/autodiscover/autodiscover.xml.

The POST request that is made by the client looks like this:

<Autodiscover xmlns="http://schemas.microsoft.com/exchange/autodiscover/outlook/requestschema/2006">
  <Request>
    <EMailAddress>probert@conatus.lan</EMailAddress>
 <AcceptableResponseSchema>http://schemas.microsoft.com/exchange/autodiscover/outlook/responseschema/2006a</AcceptableResponseSchema>
  </Request>
</Autodiscover>

One important note: this POST request is authenticated by sending the user's credentials with the Authorization header. It will use either Basic or NTLM authentication.

The response will send the details of where the different services (SMTP, EWS, POP3, IMAP, etc.) are located.

<Autodiscover xmlns="http://schemas.microsoft.com/exchange/autodiscover/responseschema/2006"> 
  <Response xmlns="http://schemas.microsoft.com/exchange/autodiscover/outlook/responseschema/2006a"> 
    <User> 
      <DisplayName>Administrator</DisplayName> 
      <EMailAddress>probert@conatus.lan</EMailAddress> 
    </User> 
    <Account> 
      <AccountType>email</AccountType> 
      <Action>settings</Action> 
      <Protocol> 
        <Type>EXCH</Type> 
        <Server>myserver.conatus.lan</Server> 
        <AD>myserver.conatus.lan</AD> 
        <ASUrl>https://myserver.conatus.lan/EWS/Exchange.asmx</ASUrl> 
        <EwsUrl>https://myserver.conatus.lan/EWS/Exchange.asmx</EwsUrl> 
        <OOFUrl>https://myserver.conatus.lan/EWS/Exchange.asmx</OOFUrl> 
      </Protocol> 
      <Protocol> 
        <Type>SMTP</Type> 
        <Server>myserver.conatus.lan</Server> 
        <LoginName>probert@conatus.lan</LoginName> 
        <Port>25</Port> 
        <DomainRequired>off</DomainRequired> 
        <SPA>off</SPA> 
        <SSL>off</SSL> 
      </Protocol> 
      <Protocol> 
        <Type>SMTP</Type> 
        <Server>myserver.conatus.lan</Server> 
        <LoginName>probert@conatus.lan</LoginName> 
        <Port>465</Port> 
        <DomainRequired>off</DomainRequired> 
        <SPA>off</SPA> 
        <SSL>on</SSL> 
      </Protocol> 
      <Protocol> 
        <Type>IMAP</Type> 
        <Server>myserver.conatus.lan</Server> 
        <LoginName>probert@conatus.lan</LoginName> 
        <Port>143</Port> 
        <DomainRequired>off</DomainRequired> 
        <SPA>off</SPA> 
        <SSL>off</SSL> 
      </Protocol> 
      <Protocol> 
        <Type>IMAP</Type> 
        <Server>myserver.conatus.lan</Server> 
        <LoginName>probert@conatus.lan</LoginName> 
        <Port>993</Port> 
        <DomainRequired>off</DomainRequired> 
        <SPA>off</SPA> 
        <SSL>on</SSL> 
      </Protocol> 
      <Protocol> 
        <Type>POP3</Type> 
        <Server>myserver.conatus.lan</Server> 
        <LoginName>probert@conatus.lan</LoginName> 
        <Port>110</Port> 
        <DomainRequired>off</DomainRequired> 
        <SPA>off</SPA> 
        <SSL>off</SSL> 
      </Protocol> 
      <Protocol> 
        <Type>POP3</Type> 
        <Server>myserver.conatus.lan</Server> 
        <LoginName>probert@conatus.lan</LoginName> 
        <Port>995</Port> 
        <DomainRequired>off</DomainRequired> 
        <SPA>off</SPA> 
        <SSL>on</SSL> 
      </Protocol> 
    </Account> 
    <Action> 
      <Settings> 
        <Server> 
          <Type>MobileSync</Type> 
          <Url>https://myserver.conatus.lan/Microsoft-Server-ActiveSync/</Url> 
          <Name>https://myserver.conatus.lan/Microsoft-Server-ActiveSync/</Name>
        </Server> 
      </Settings> 
    </Action> 
  </Response> 
</Autodiscover>

Once it located the SOAP endpoint for EWS (the value of the EwsUrl attribute in the response above), the client can fetch the folders and everything else.

Network/HTTP debugging tools

Since I'm working on CalDAV/CardDAV, Exchange Web Services and Zimbra SOAP APIs, I often need to see what's happening on the wire. To achieve this, I use a couple of tools:

• tcpflow (installed with MacPorts). tcpflow allows use to see TCP trafic, on a specific network interface. You can use rules to specific which host(s) or port(s) that you want to sniff, which is quite useful. But since it log the packets, if the data is encrypted, you can't see unencrypted.
• Charles Proxy. Since I really need to see the unencrypted data (my Exchange hosting provider only allow HTTPS), I'm using Charles to act as a HTTP/HTTPS proxy. Charles is really great, it will show SOAP or REST request/response in a nice layout, and the greatest thing: it can act as a HTTPS proxy, so you can see the unencrypted data! The only downside is that I wasn't able to get it to work with Outlook 2011 and a HTTPS Exchange server (it will always return a 401 Not Authorized response).
• Debug window in Zimbra. You can use some query arguments when logging into Zimbra Web Client, arguments that will allow use to see the SOAP requests that the client is sending to the back-end. Very useful when looking for values that Zimbra is sending.
• Debug mode in iCal. If you don't want to use a proxy, you can see what's iCal (or Calendar on 10.8) by enabling the log for HTTP activity made by iCal. To do so, just run:
  
  defaults write com.apple.iCal LogHTTPActivity -boolean TRUE

Update: a friend on Google+ sent me a link to a Wireshark tutorial. I must admit that I prefer tcpflow for low level debugging, but Wireshark and tcpdump are indeed other kind of tools who can use (lsof is also a tool that I used to find what applications and services are using for networking and file handles).

CrashPlan not working with Java Update 1.06.0_37

Got hit by the Java 1.7/CrashPlan problem that Java Update 1.06.0_37 creates. If you installed that update and you have Java 1.7 from Oracle on your Mac, you will need to follow this procedure.

Outlook Web Access and existing folders...

I'm working on code to create folders with Exchange Web Services (request is done by a Java API). Since I'm testing, I sometimes trash a new folder and recreate a folder with the same name as the first folder that I deleted.

But if I try to delete the new folder, Outlook Web Access (OWA) tells me that a folder with the same name already exists in the Deleted Items, so I can't delete the new folder... I have to empty the Deleted Items folder to be able to delete the folder I re-created...

CalDAV standards and extensions

The Calendaring Standards page on CalConnect.org have a list of RFCs and drafts that CalDAV implementations uses. The main RFC is 4791, all CalDAV servers must implement this RFC.

Sadly, the page on CalConnect is missing a couple of references.

• caldav-sharing
• caldav-notifications
• draft-daboo-carddav-directory-gateway

Those three extensions have been implemented in iCal Server and Calendar Server, so it's a good thing to learn them.

Update: the missing references have been added to the Calendaring Standards page.

If you wish to see which CalDAV servers is using extensions and new RFCs, I have built a (incomplete) list of CalDAV implementations and match them with the features found in the RFCs and extensions.

CalDAV handsake

When you add an CalDAV account in iCal (OS X 10.7, aka Lion, or earlier) or in Calendar (OS X 10.8, aka Mountain Lion), it does a lot of magic to configure the account. In this post, I will explain how the "handsake" is done, and what kind of data and queries is done between the client and the server. From now on, I will use iCal as the name of the client, but everything is valid for the Calendar app in 10.8 too.

When you set the account type to Automatic, as explained in my previous blog post, iCal will do a DNS query to find out what is the host name of the CalDAV server. If a SRV DNS entry have been found, iCal will get the hostname and TCP port from the SRV entry, and will do the following (here, I'm assuming that the CalDAV server found in the SRV entry is "server.macti.lan" and the TCP port is 8008):

• Do a PROPFIND (WebDAV method) request on http://server.macti.lan:80/.well-known/caldav with the following body:

  <?xml version="1.0" encoding="UTF-8"?>
  <A:propfind xmlns:A="DAV:">
    <A:prop>
      <A:current-user-principal/>
      <A:principal-URL/>
      <A:resourcetype/>
    </A:prop>
  </A:propfind>

• If iCal didn't get a valid response on port 80, it will try the same request, but on port 8080 instead.
• If iCal didn't get a valid response on port 80 or 8080, it will try on the port it found in the SRV entry, in our case, port 8008.

The response from /.well-known/caldav might be a redirect (HTTP status code 301 or 302). In fact, that's what iCal Server 10.8 does, it redirect the request to http://server.macti.lan:8008/. So again, iCal will do the same request as before until it gets a 207 status code instead of a redirect. The response will look like this:

<?xml version='1.0' encoding='UTF-8'?>
<multistatus xmlns='DAV:'>
  <response>
    <href>/</href>
    <propstat>
      <prop>
        <current-user-principal>
          <href>/principals/__uids__/C0F07FAF-A20A-4A88-A518-D27E5D3074CA/</href>
        </current-user-principal>
        <resourcetype>
          <collection/>
        </resourcetype>
      </prop>
      <status>HTTP/1.1 200 OK</status>
    </propstat>
  </response>
</multistatus>

The value we need is the value of current-user-principal, which indicates the location of the user's "principals". With this value, iCal will make a OPTIONS request to /principals/__uids__/C0F07FAF-A20A-4A88-A518-D27E5D3074CA/. I don't know why iCal is making the request, but my guess it's doing it so that it can find which CalDAV features are supported by the CalDAV server. That list is available in the response's DAV header:

DAV: 1, access-control, calendar-access, calendar-schedule, calendar-auto-schedule, calendar-availability, inbox-availability, calendar-proxy, calendarserver-private-events, calendarserver-private-comments, calendarserver-sharing, calendarserver-sharing-no-scheduling, calendar-query-extended, calendar-default-alarms, addressbook, extended-mkcol, calendarserver-principal-property-search

This is a response from iCal Server 10.8, which supports all CalDAV features and extensions. Doing the same request to a Google Calendar server will return a lot less features/extensions.

Following the OPTIONS request, iCal will make a PROPFIND request on the same URL as the OPTIONS request, with the following body:

<?xml version="1.0" encoding="UTF-8"?>
<A:propfind xmlns:A="DAV:">
  <A:prop>
    <C:calendar-home-set xmlns:C="urn:ietf:params:xml:ns:caldav"/>
    <C:calendar-user-address-set xmlns:C="urn:ietf:params:xml:ns:caldav"/>
    <A:current-user-principal/>
    <A:displayname/>
    <B:dropbox-home-URL xmlns:B="http://calendarserver.org/ns/"/>
    <B:email-address-set xmlns:B="http://calendarserver.org/ns/"/>
    <B:notification-URL xmlns:B="http://calendarserver.org/ns/"/>
    <A:principal-collection-set/>
    <A:principal-URL/>
    <A:resource-id/>
    <C:schedule-inbox-URL xmlns:C="urn:ietf:params:xml:ns:caldav"/&gt
    <C:schedule-outbox-URL xmlns:C="urn:ietf:params:xml:ns:caldav"/>    <A:supported-report-set/>
  </A:prop>
</A:propfind>

The value that we need back from this request is the value of the calendar-home-set property. This property is the URL to calendar collections. So iCal can finally found where the calendars are, and it will do a PROPFIND request to the URL found with the value of calendar-home-set. The body of that PROPFIND request is quite large so I'm not going to past it here, but in short, that PROPFIND will find the request properties for each sub-collections in the main calendar collection. For example, if the main collection is located at:

/calendars/__uids__/0B9B4FA7-8ADC-4984-8258-D04A4939A574/

The "calendar" collection will be located at:

/calendars/__uids__/0B9B4FA7-8ADC-4984-8258-D04A4939A574/calendar/ 

And the "tasks" collection:

/calendars/__uids__/0B9B4FA7-8ADC-4984-8258-D04A4939A574/tasks

So, for each sub-collection, iCal will do the following:

• A PROPFIND request to find the value of the checksum-versions property;
• A PROPFIND request to find the value of the getctag and sync-token properties;
• A PROPFIND request to find the value of the getcontenttype and getetag properties;

The third request, who fetches the content type and etag, is the one that will return a link to all iCalendar objects from the calendar collection. The response will return a <response> XML attribute for each iCalendar objects, with the following structure:

  <response>
    <href>/calendars/__uids__/C0F07FAF-A20A-4A88-A518-D27E5D3074CA/calendar/20111210T013923Z-uidGen%40mbp-pascal-robert-4.local.ics</href>
    <propstat>
      <prop>
        <getcontenttype>text/calendar;charset=utf-8</getcontenttype>
        <getetag>"9b6b2d11f86891f748ef09ab0993b75c"</getetag>
      </prop>
      <status>HTTP/1.1 200 OK</status>
    </propstat>
  </response>

From there, iCal will simply fetch all valid iCalendar objects and display them in the calendar window. This is done by doing a REPORT request on the calendar collection, with the calendar-multiget attribute, with the list of calendar objects that the previous PROPFIND response have returned.

<?xml version="1.0" encoding="UTF-8"?><B:calendar-multiget xmlns:B="urn:ietf:params:xml:ns:caldav">  <A:prop xmlns:A="DAV:">    <A:getetag/>    <B:calendar-data/>    <C:updated-by xmlns:C="http://calendarserver.org/ns/"/>    <C:created-by xmlns:C="http://calendarserver.org/ns/"/>  </A:prop>  <A:href xmlns:A="DAV:">/calendars/__uids__/C0F07FAF-A20A-4A88-A518-D27E5D3074CA/calendar/20111210T013923Z-uidGen%40mbp-pascal-robert-4.local.ics</A:href></B:calendar-multiget> 

This is the response:

<D:multistatus xmlns:D="DAV:">  <D:response>  <D:href>/calendars/__uids__/C0F07FAF-A20A-4A88-A518-D27E5D3074CA/calendar/20111210T013923Z-uidGen%40mbp-pascal-robert-4.local.ics</D:href>  <D:propstat>  <D:status>HTTP/1.1 200 OK</D:status>  <D:prop>  <D:getetag>"63459166486"</D:getetag>  <C:calendar-data xmlns:C="urn:ietf:params:xml:ns:caldav"> BEGIN:VCALENDAR PRODID:-//Google Inc//Google Calendar 70.9054//EN VERSION:2.0 CALSCALE:GREGORIAN X-WR-CALNAME:Pascal Robert X-WR-TIMEZONE:America/New_York  BEGIN:VEVENT DTSTART:20111209T140000Z  DTEND:20111209T180000Z DTSTAMP:20111210T021446Z  ORGANIZER;CN=probert@macti.lan:mailto:probert@macti.lan UID:20111210T021447Z-uidGen@mbp-pascal-robert-4.local  ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;CN=Pascal Robert;X-NUM-GUESTS=0:mailto:pascal.probert@gmail.com  CLASS:CONFIDENTIAL CREATED:20111210T021446Z  DESCRIPTION:Un plus long texte LAST-MODIFIED:20111210T021446Z  LOCATION:Un endroit SEQUENCE:0  STATUS:CONFIRMED SUMMARY:Événement test  TRANSP:OPAQUE BEGIN:VALARM ACTION:DISPLAY DESCRIPTION:This is an event reminder  TRIGGER:-P0DT0H30M0S END:VALARM BEGIN:VALARM ACTION:AUDIO TRIGGER;VALUE=DATE-TIME:20111209T134500Z END:VALARM END:VEVENT END:VCALENDAR </C:calendar-data>  </D:prop>  </D:propstat>  <D:propstat>  <D:status>HTTP/1.1 404 Not Found</D:status>  <D:prop>  <C:updated-by xmlns:C="http://calendarserver.org/ns/" />  <C:created-by xmlns:C="http://calendarserver.org/ns/" />  </D:prop>  </D:propstat>  </D:response> </D:multistatus>

CalDAV/CardDAV discovery

If you use iCal/Calendar on Mac OS X or iOS, you might have seen that you don't have to specify the type of server ("Automatic" is the default type). But to work, you need to setup DNS entries so that iCal/Calendar can find the server for the user, and your CalDAV server must support "well-known" URLs. "well-known" is supported by iCal Server (OS X Server 10.7 or 10.8), Calendar Server (the open source version of iCal Server) and Kerio Connect (starting with version 7.5 I believe).

For DNS, I found how to do it in this article. In short, you need at least one SRV entry that will tell where your server is located. If your domain is "macti.lan", you need to add the following entry:

_caldavs._tcp.macti.lan.            10800 IN SRV      10 1 8443 server.macti.lan.

Where 8443 is the port on which your CalDAV server is running (8443 is the HTTPS port for iCal Server/Calendar Server, for Kerio Connect by default it's on port 443). server.macti.lan is the DNS name of the host running the CalDAV service.

You will notice that the DNS entry is _caldavs_. If your CalDAV server is running on a non-protected port, or if it's available on both ports, you should add a DNS entry like this:

_caldav._tcp.macti.lan.            10800 IN SRV      10 1 8008 server.macti.lan.

Notice that the 's' is missing in the DNS entry. If you have both entries, iCal/Calendar will try the secure entry first, and it's not working, it will try the non-secure entry after.

If you have a iCloud account with a @me.com, a SRV entry do exist:

$ dig _caldavs._tcp.me.com srv 
_caldavs._tcp.me.com. 3600 IN SRV 0 0 443 caldav.icloud.com.