Creating SSL certificates for web, mail and IM servers is not well documented. Here are some notes on generating certificates on a Debian server, signed by your own certificate authority, plus instructions for using them on OS X (instructions for installing and configuring openssl, apache2 and mod_ssl, exim4, courier-imap-ssl, courier-pop3-ssl and jabberd2 are not included):

Certificate Authority

First of all you need to create a root Certificate Authority certificate [ca.crt], which will later be used to sign the other certificates. It's encrypted by a password-protected key [ca.key].

cd /usr/share/ssl-cert/

Create index and serial files

cp /dev/null ca.index
echo '01' >ca.serial

Edit file paths in the default config file

[here's mine]

nano ca.config

Generate key and certificate for signing authority

openssl genrsa -des3 -out ca.key 2048
openssl req -new -x509 -days 3650 -key ca.key -out ca.crt

(use . for everything except the Common Name, which is the name of your Certificate Authority, eg Example Certificate Authority)

So that OS X trusts all certificates signed by your Certificate Authority, download ca.crt then double-click and import it into the X509Anchors keychain using Keychain Access.

Apache2

For Apache, a key [example.org.key] is generated and used to encrypt a certificate request [example.org.csr], which is then signed by the Certificate Authority to produce the final certificate [example.org.crt]. The files are named this way so that you can have multiple certificates for virtual hosts. The keys must not be password protected, otherwise Apache will hang waiting for a password when it starts up.

Generate key and certificate for Apache

openssl genrsa -out example.org.key 2048
openssl req -new -key example.org.key -out example.org.csr

(use . for everything except the Common Name, which is the web server address, eg www.example.org)

Sign Apache certificate

openssl ca -config ca.config -keyfile ca.key -cert ca.crt -out example.org.crt -infiles example.org.csr

Set up Apache

cp example.org.key /etc/apache2/ssl/
cp example.org.crt /etc/apache2/ssl/

nano /etc/apache2/sites-enabled/example.org
<VirtualHost *:443>
ServerName www.example.org

SSLEngine on
SSLCertificateFile /etc/apache2/ssl/example.org.crt
SSLCertificateKeyFile /etc/apache2/ssl/example.org.key
</VirtualHost>

apache2ctl stop
apache2ctl start

Courier

For Courier, a non-protected key [courier.key] is generated and used to encrypt a certificate request [courier.csr], which is then signed by the Certificate Authority to produce the final certificate [courier.crt]. The key and certificate are combined into a PEM file [courier.pem], which is then used for both the IMAP and POP3 servers.

Generate key and certificate for Courier

openssl genrsa -out courier.key 2048
openssl req -new -key courier.key -out courier.csr

(use . for everything except the Common Name, which is the mailserver address, eg mail.example.org)

Sign Courier certificate

openssl ca -config ca.config -keyfile ca.key -cert ca.crt -out courier.crt -infiles courier.csr

Combine key and certificate into PEM file

nano courier.crt

(strip out eveything before the BEGIN CERTIFICATE line)

cat courier.key courier.crt > courier.pem
openssl gendh >> courier.pem

Set up Courier

cp courier.pem /etc/courier/imapd.pem
cp courier.pem /etc/courier/pop3d.pem
chmod 0600 /etc/courier/imapd.pem
chmod 0600 /etc/courier/pop3d.pem
/etc/init.d/courier-imap-ssl stop
/etc/init.d/courier-imap-ssl start
/etc/init.d/courier-pop-ssl stop
/etc/init.d/courier-pop-ssl start

Exim

For Exim4, a non-protected key [exim.key] and certificate request [exim.csr] are generated. The request is then signed by the Certificate Authority to produce the final certificate [exim.crt].

Generate key and certificate for Exim4

openssl req -newkey rsa:2048 -keyout exim.key -out exim.csr -days 3650 -nodes

(use . for everything except the Common Name, which is the SMTP server address, eg smtp.example.org)

Sign Exim4 certificate

openssl ca -config ca.config -keyfile ca.key -cert ca.crt -out exim.crt -infiles exim.csr

Set up Exim4

cp exim.crt /etc/exim4/exim.crt
cp exim.key /etc/exim4/exim.key
chown root:Debian-exim /etc/exim4/exim.key /etc/exim4/exim.crt
chmod 640 /etc/exim4/exim.key /etc/exim4/exim.crt
/etc/init.d/exim4 stop
/etc/init.d/exim4 start

Jabberd2

For Jabberd2, a non-protected key [jabberd.key] is generated and used to encrypt a certificate request [jabberd.csr], which is then signed by the Certificate Authority to produce the final certificate [jabberd.crt]. The key and certificate are combined into a PEM file [jabberd.pem], which is used for messages between client and server.

Generate key and certificate for Jabberd

openssl genrsa -out jabberd.key 2048
openssl req -new -key jabberd.key -out jabberd.csr

(use . for everything except the Common Name, which is the Jabber server address, eg jabber.example.org)

Sign Jabberd certificate

openssl ca -config ca.config -keyfile ca.key -cert ca.crt -out jabberd.crt -infiles jabberd.csr

Combine key and certificate into PEM file

nano jabberd.crt

(strip out eveything before BEGIN CERTIFICATE)

cat jabberd.key jabberd.crt > jabberd.pem

Set up Jabberd

cp jabberd.pem /usr/local/etc/jabberd/
chown root:jabber /usr/local/etc/jabberd/jabberd.pem
chmod 640 /usr/local/etc/jabberd/jabberd.pem

( edit all xml files so that points to /usr/local/etc/jabberd/jabberd.pem )

su jabber
jabberd &

Psi is the only OS X Jabber client (as far as I know) that verifies the authenticity of SSL certificates, though there is an option to hide warnings. It comes with a set of root certificates, to which you have to add the certificate for your root Certificate Authority (instead of using the system Keychain).

In ca.crt, replace

-----BEGIN CERTIFICATE-----

with

<certificate><data>

and

-----END CERTIFICATE-----

with

</data></certificate>

Do Show Package Contents on Psi.app, then open Contents/Resources/certs/rootcert.xml and add in the new data from ca.crt. Connect to the server on port 5223.