Creating SSL certificates for web, mail and IM servers is not well documented. Here are some notes on generating certificates on a Debian server, signed by your own certificate authority, plus instructions for using them on OS X (instructions for installing and configuring openssl, apache2 and mod_ssl, exim4, courier-imap-ssl, courier-pop3-ssl and jabberd2 are not included):
Certificate Authority
First of all you need to create a root Certificate Authority certificate [ca.crt], which will later be used to sign the other certificates. It's encrypted by a password-protected key [ca.key].
cd /usr/share/ssl-cert/
Create index and serial files
cp /dev/null ca.index
echo '01' >ca.serial
Edit file paths in the default config file
nano ca.config
Generate key and certificate for signing authority
openssl genrsa -des3 -out ca.key 2048
openssl req -new -x509 -days 3650 -key ca.key -out ca.crt
(use . for everything except the Common Name, which is the name of your Certificate Authority, eg Example Certificate Authority)
So that OS X trusts all certificates signed by your Certificate Authority, download ca.crt then double-click and import it into the X509Anchors keychain using Keychain Access.
Apache2
For Apache, a key [example.org.key] is generated and used to encrypt a certificate request [example.org.csr], which is then signed by the Certificate Authority to produce the final certificate [example.org.crt]. The files are named this way so that you can have multiple certificates for virtual hosts. The keys must not be password protected, otherwise Apache will hang waiting for a password when it starts up.
Generate key and certificate for Apache
openssl genrsa -out example.org.key 2048
openssl req -new -key example.org.key -out example.org.csr
(use . for everything except the Common Name, which is the web server address, eg www.example.org)
Sign Apache certificate
openssl ca -config ca.config -keyfile ca.key -cert ca.crt -out example.org.crt -infiles example.org.csr
Set up Apache
cp example.org.key /etc/apache2/ssl/
cp example.org.crt /etc/apache2/ssl/
nano /etc/apache2/sites-enabled/example.org
<VirtualHost *:443>
ServerName www.example.org
SSLEngine on
SSLCertificateFile /etc/apache2/ssl/example.org.crt
SSLCertificateKeyFile /etc/apache2/ssl/example.org.key
</VirtualHost>
apache2ctl stop
apache2ctl start
Courier
For Courier, a non-protected key [courier.key] is generated and used to encrypt a certificate request [courier.csr], which is then signed by the Certificate Authority to produce the final certificate [courier.crt]. The key and certificate are combined into a PEM file [courier.pem], which is then used for both the IMAP and POP3 servers.
Generate key and certificate for Courier
openssl genrsa -out courier.key 2048
openssl req -new -key courier.key -out courier.csr
(use . for everything except the Common Name, which is the mailserver address, eg mail.example.org)
Sign Courier certificate
openssl ca -config ca.config -keyfile ca.key -cert ca.crt -out courier.crt -infiles courier.csr
Combine key and certificate into PEM file
nano courier.crt
(strip out eveything before the BEGIN CERTIFICATE line)
cat courier.key courier.crt > courier.pem
openssl gendh >> courier.pem
Set up Courier
cp courier.pem /etc/courier/imapd.pem
cp courier.pem /etc/courier/pop3d.pem
chmod 0600 /etc/courier/imapd.pem
chmod 0600 /etc/courier/pop3d.pem
/etc/init.d/courier-imap-ssl stop
/etc/init.d/courier-imap-ssl start
/etc/init.d/courier-pop-ssl stop
/etc/init.d/courier-pop-ssl start
Exim
For Exim4, a non-protected key [exim.key] and certificate request [exim.csr] are generated. The request is then signed by the Certificate Authority to produce the final certificate [exim.crt].
Generate key and certificate for Exim4
openssl req -newkey rsa:2048 -keyout exim.key -out exim.csr -days 3650 -nodes
(use . for everything except the Common Name, which is the SMTP server address, eg smtp.example.org)
Sign Exim4 certificate
openssl ca -config ca.config -keyfile ca.key -cert ca.crt -out exim.crt -infiles exim.csr
Set up Exim4
cp exim.crt /etc/exim4/exim.crt
cp exim.key /etc/exim4/exim.key
chown root:Debian-exim /etc/exim4/exim.key /etc/exim4/exim.crt
chmod 640 /etc/exim4/exim.key /etc/exim4/exim.crt
/etc/init.d/exim4 stop
/etc/init.d/exim4 start
Jabberd2
For Jabberd2, a non-protected key [jabberd.key] is generated and used to encrypt a certificate request [jabberd.csr], which is then signed by the Certificate Authority to produce the final certificate [jabberd.crt]. The key and certificate are combined into a PEM file [jabberd.pem], which is used for messages between client and server.
Generate key and certificate for Jabberd
openssl genrsa -out jabberd.key 2048
openssl req -new -key jabberd.key -out jabberd.csr
(use . for everything except the Common Name, which is the Jabber server address, eg jabber.example.org)
Sign Jabberd certificate
openssl ca -config ca.config -keyfile ca.key -cert ca.crt -out jabberd.crt -infiles jabberd.csr
Combine key and certificate into PEM file
nano jabberd.crt
(strip out eveything before BEGIN CERTIFICATE)
cat jabberd.key jabberd.crt > jabberd.pem
Set up Jabberd
cp jabberd.pem /usr/local/etc/jabberd/
chown root:jabber /usr/local/etc/jabberd/jabberd.pem
chmod 640 /usr/local/etc/jabberd/jabberd.pem
( edit all xml files so that points to /usr/local/etc/jabberd/jabberd.pem )
su jabber
jabberd &
Psi is the only OS X Jabber client (as far as I know) that verifies the authenticity of SSL certificates, though there is an option to hide warnings. It comes with a set of root certificates, to which you have to add the certificate for your root Certificate Authority (instead of using the system Keychain).
In ca.crt, replace
-----BEGIN CERTIFICATE-----
with
<certificate><data>
and
-----END CERTIFICATE-----
with
</data></certificate>
Do Show Package Contents on Psi.app, then open Contents/Resources/certs/rootcert.xml and add in the new data from ca.crt. Connect to the server on port 5223.