How to secure a mail server using encryption

SSL (Secure Sockets Layer) and its descendant TLS (Transport Layer Security) are the most widely used protocols for encrypting data that is exchanged between a server and a client. These protocols often use X.509 certificates and asymmetric cryptography.

STARTTTLS is another method of securing plain-text communication. This protocol also encrypts data with SSL or TLS, but with the same port as the plain-text protocols, instead of using separate ports for SSL/TLS-encrypted communications. For example, IMAP over STARTTLS uses the same port as IMAP (143), while IMAPS (IMAP over SSL) uses a separate port 993.

The previous tutorial describes how to set up a mail server running on Postfix and Dovecot, but the security aspect was not covered. In this tutorial, we demonstrate how to add security to a mail server through TLS/SSL-based encryption.

Certificates needed for TLS/SSL can be self-signed, signed by a free certification authority (e.g., CAcert) or signed by a commercial authority (e.g., VeriSign), and can be generated with utilities like OpenSSL. We are going to use a self-signed certificate in this tutorial.

Enable TLS Encryption for Postfix

A self-signed certificate can be created with the following command.

# openssl req -new -x509 -days 365 -nodes -out /etc/ssl/certs/postfixcert.pem -keyout /etc/ssl/private/postfixkey.pem

The above command requests a new certificate which is of type X.509, and remains valid for 365 days. The optional -nodes parameter specifies that the private key should not be encrypted. An output certificate file is saved as postfixcert.pem, and an output key file as postfixkey.pem .

All necessary values for the certificate can be given:

Country Name (2 letter code) [AU]:BD 
State or Province Name (full name) [Some-State]:Dhaka 
Locality Name (eg, city) []:Dhaka 
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:Example.tst 
Common Name (e.g. server FQDN or YOUR name) []:mail.example.tst 
Email Address []:sarmed@example.tst 

Now that the certificate is ready, necessary parameters are adjusted in postfix configuration file.

root@mail:~# vim /etc/postfix/
### STARTTLS is enabled ###
smtpd_tls_security_level = may 

smtpd_tls_received_header = yes 
smtpd_tls_auth_only = yes 

### loglevel 3 should be used while troubleshooting ###
smtpd_tls_loglevel = 1

### path to certificate and key file
smtpd_tls_cert_file = /etc/ssl/certs/postfixcert.pem 
smtpd_tls_key_file = /etc/ssl/private/postfixkey.pem 

Restart postfix to enable TLS.

root@mail:~# service postfix restart

At this point, postfix is ready to encrypt data to and from the server. More details about Postfix TLS support can be found in their official README.

Enable SSL Encryption for Dovecot

Configuring dovecot for encryption is similar to postfix.

First of all, a self-signed certificate is created with openssl:

# openssl req -new -x509 -days 365 -nodes -out /etc/ssl/certs/dovecotcert.pem -keyout /etc/ssl/private/dovecotkey.pem

The above command requests a new X.509 certificate which is valid for 365 days. -nodes is an optional parameter which specifies that the stored private key should not be encrypted. An output certificate file will be dovecotcert.pem, and an output key file will be dovecotkey.pem.

All necessary parameters need to be specified in the certificate:

Country Name (2 letter code) [AU]:BD
State or Province Name (full name) [Some-State]:Dhaka
Locality Name (eg, city) []:Dhaka
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:Example.tst
Common Name (e.g. server FQDN or YOUR name) []:mail.example.tst
Email Address []:sarmed@example.tst

Next, the path to the certificate is added in dovecot configuration.

root@mail:~# vim /etc/dovecot/conf.d/10-ssl.conf
ssl_cert = </etc/ssl/certs/dovecotcert.pem
ssl_key = </etc/ssl/private/dovecotkey.pem

Finally, dovecot is restarted to enable SSL with the new certificate.

root@mail:~# service dovecot restart

Thunderbird Mail Client Configuration

The following is a snapshot on how to configure the account in Mozilla Thunderbird.


First of all, make sure that all necessary ports are allowed in the firewall.

Second, try telnet to a mail server. You should be able to get through. Some examples are given below for reference.

Connect to IMAPS

$ telnet mail.example.tst 993
Trying mail.example.tst... 
Connected to mail.example.tst. 
Escape character is '^]'. 
Connection closed by foreign host. 

Connect to POP3S

$ telnet mail.example.tst 995
Trying mail.example.tst... 
Connected to mail.example.tst. 
Escape character is '^]'. 
Connection closed by foreign host. 

Connect to SMTP

$ telnet mail.example.tst 25
Trying mail.example.tst... 
Connected to mail.example.tst. 
Escape character is '^]'. 
220 mail.example.tst ESMTP Postfix (Ubuntu) 

### Command ###
ehlo mail.example.tst 
250-SIZE 10240000 
250 DSN

Subscribe to Xmodulo

Do you want to receive Linux FAQs, detailed tutorials and tips published at Xmodulo? Enter your email address below, and we will deliver our Linux posts straight to your email box, for free. Delivery powered by Google Feedburner.

Support Xmodulo

Did you find this tutorial helpful? Then please be generous and support Xmodulo!

The following two tabs change content below.
Sarmed Rahman is an IT professional based in Australia. He writes tutorial articles on technology every now and then from a belief that knowledge grows through sharing. During his free time, he loves gaming and spending time with his friends.

7 thoughts on “How to secure a mail server using encryption

  1. This isn't securing a mail server, it's securing *client connections* with a mail server using TLS.

    Well-written article, mind: covers changes to mail server, mail transport, cert generation AND testing with the client. You could possibly drop squirrelmail config (or maybe Outlook) into that just to show another cilent making use of encrypted connections.

    I had to jump through these same hoops a number of years ago (I use IMAPs with TBird) and ended up ditching Dovecot when it broke on me twice - both times when I was in the states. I switched to Courier and it's been running perfectly fine every since. Nothing against Dovecot or your article, just once bitten ...

    • Thank you for sharing. Yes, this tutorial focuses more on securing the client-server communication with use of encryption.

      Personally, some of the dovecot servers that I have deployed are running more than 3 years now. So far so good. Still, could you please share what trouble you faced with dovecot and it was solved so we all can be prepared for the future?

      Thanks again.

      • Well, to expand on what I said up there...

        I had it all working fine, IMAPs and POP3s all talking to my server, then a quick "yum update" brought a newer version down which included some bugs - forums were rife with others encountering the same issue and a week later a hastily-released update fixed the issues. This actually occurred to be when abroad (states-side) and I was using squirrelmail to access my mail; I ended up reading my mbox spool over ssh to check last few incoming mails.

        I couldn't roll back, so uninstalled the latest version and used an older version in /var/cache/yum and all worked. Later I heard of the fix so yum-update did its thing and all was well.

        Two years later, I was abroad again (same country, same house) and Dovecot broke again - almost as though it was awaiting the worse convenient moment. Unfortunately a "yum clean cache" had flushed out earlier versions and I couldn't locate an older RPM to use, so I just awaited until a patched version came out. From that point I decided I wasn't going to trust it... I looked into Courier, jumped through a few hoops messing around with saslauthd and converting mbox to Maildir... and it's been working all fine since.

        Now granted this was on Fedora Core 4 years ago - at the time when Fedora was jokingly remarked as being the betatest ground for RHEL - so I was willing to put up with experimental oddities failing here and there, but I felt Dovecot has sadly let me down once too many.

        I don't have any issues with giving it another go, but I have a working model with Courier so I'll stick to that.

        • So basically you did not read the changelog of the new Dovecot version or the feedback from other Mensa members like you, you upgraded without first testing the new version, directly in your production environment and you go around bitching about Dovecot?

          Ha ha ha. If you weren't so pathetic you would be funny.

          Dovecot rules - in terms of speed, memory footprint, etc. I have been serving about more than 150000 mailboxes for five years now without a single problem. Not one.

          But then I don't go around shooting myself in the leg like you.

  2. Nice article, short and concise.

    I just want to suggest three things.

    1) Show how to generate a real certificate from CACert or StartSSL. Two gratis certificat authorities that work well. CACert works with any domain you have controll over, and StartSSL only with top domain, so works, but not

    2) How to make it work with IPv6.

    3) Use the standard example domains and ip-addresses in your examples. There are some RFCs you should use, like RFC 5737, RFC 3849 and RFC 6761. Please try to start from and you get some more information.

    But I like the form of these HOWTOs.

    Thank you.

Leave a comment

Your email address will not be published. Required fields are marked *